Multimedia Beta-version EjUAnPc2VA#t=20.

125
CS338 Multimedia Beta-version http://www.youtube.com/watch_p opup?v=jEjUAnPc2VA#t=20
  • date post

    19-Dec-2015
  • Category

    Documents

  • view

    216
  • download

    1

Transcript of Multimedia Beta-version EjUAnPc2VA#t=20.

CS338

MultimediaBeta-version

httpwwwyoutubecomwatch_popupv=jEjUAnPc2VAt=20

Overview -1

Required background C C++ or Java CS240 (Data Structures) or CS212 (for

Engineers) This is NOT a graphics course (see

CS460560) Labs

Data compression Elements of graphics in OpenGL Photo manipulation

GIMP Steganography

Creating amp Rendering modelscopy DJ Foreman 2012 2

Topics (general order of coverage)

Compression EncodingDecoding Codecs Huffman encoding

example lab Working with Images

Image creation Photography Drawing RIB (Renderman Interface

Bytestream) Modeling

OpenGL ndash lab Makehuman 3DCanvas etc

lab Image manipulation

Gimp Photoshop etc lab Rendering Bitmaps

Ray tracing

HTML CSS amp Javascript - lab

Animation Hand drawn Program generated

Stop-motion Alice lab DirectX amp OpenGL

Hardware acceleration RIB revisited lab Shaders

Steganography lab Additional topics (time

permitting) Augmented reality Geospatial data systems

copy DJ Foreman 2012 3

Software we might use (all free)

Modelers and environments Ayam ndash ortho amp 3D 3DCanvas ndash 3D Blender ndash ortho amp 3D Makehuman - 3D

Renderers Aqsis Pixie ndash no GUI file input

copy DJ Foreman 2012 4

5

Image formats

Bitmap Also called ldquoraster imagesrdquo

Stored as individual pixels Resolution dependent Screen resolution = 100 ppi Printing requires 150-300 ppi

Vector Scalable primitives amp formulas Resolution independent

copy DJ Foreman 2012

6

Bitmap formats

BMP (program generated) GIF (ldquo but copy limits on use of format) JPEG JPG (photo) PNG (like GIF wo copy limits) PICT(ldquo Mac BMP+Vector) PCX (ldquo Personal Computer Exchange) PSD (Photoshop) TIFF (ldquo can also be a container)

copy DJ Foreman 2012

7

Vector formats

AI (Adobe Illustrator) CDR (CorelDRAW) CMX (Corel Exchange) CGM (Computer Graphics Metafile ) DXF (AutoCAD ) WMF (Windows Metafile )

copy DJ Foreman 2012

8copy DJ Foreman 2012

Compression

Codecs A device or program for encoding andor

decoding a digital data stream or signal Encoding A-gtD for storage or transmission DecodingD-gtA for playback Lossy vs lossless Raw uncompressed Pulse Code Modulation

(PCM) audio a digital representation of an analog signal where the magnitude of the signal is sampled regularly at uniform intervals Eg (441 kHz 16 bit stereo as represented on an

audio CD or in a wav or aiff file) is a standard across multiple platforms

copy DJ Foreman 2012 9

Codecs - 2

May emphasize aspects of data Video

motion vs color Audio

latency (cell phones) vs high-fidelity (music) Data size

transmission speed or storage size vs data loss

copy DJ Foreman 2012 10

Codec References

Forensic Computing A Practitioners Guide by T Sammes amp B Jenkinson (Springer 2000) ISBN 9781852332990

httpwwwgarykesslernetlibraryfile_sigshtml

copy DJ Foreman 2012 11

Considerations

Tradeoffs compression speed compressed data size quality (data loss)

Areas of study information theory rate-distortion theory

Popular algorithms (all lossless) Lempel-Ziv (LZ) LZW (fast decompression) LZR (LZ-Renau) (ZIP files) LZW (Lempel-Ziv-Welch) (for GIF files)

copy DJ Foreman 2012 12

Lossless Encoding

Huffman compression Bit string representing any symbol is never a

prefix of the bit string representing any other symbolie if 010 is the code for a symbol then no other symbol starts with 010

Frequency table must be known computable or included

Lossless ndash every character gets encoded Arithmetic encoding

Probabilistic algorithm Slightly superior to Huffman but often

patentedcopy DJ Foreman 2012 13

Methodology

Compute a model of the data eg a Huffman tree based on probability Probability determined from input file

Map the data according to the model Read 1 symbol Search the model (tree) for that symbol

If itrsquos a Huffman tree going left=0 right=1 Append each 0 or 1 to the code string When symbol is found you are done Note all symbols will have lt 8 bits (thus

compression)

copy DJ Foreman 2012 14

15

Huffman lab ndash more details Read write are always in bytes

You may need to pad You may have extraneous letters in the output

due to padding You donrsquot need to add a header to specify size

You have 2 weeks to do the lab Code may be in C C++ or Java The readme specifies how to compile the

program Encoded data is sent to a binary file or a simple

text file (but isnrsquot ldquoreadablerdquo as text) Encoded data can only be ldquocheckedrdquo with a

decodercopy DJ Foreman 2012

16

Sample of prefix coding amp compression

copy DJ Foreman 2012

B e a r c a t s (8 bytes of text)010101100110001110001111 encoded as123456781234567812345678 3 bytes

A simple Huffman-like code(wo probability basis) 0 1

A 110

B 010

C 0001

E 10

R 011

S 111

T 001

Compression-Model types

Static Read all the data Compute frequencies Re-read the data amp encode

Dynamic Build simple basic model Modify model as more data is processed

copy DJ Foreman 2012 17

Standard Compressed-File Content

Three parts 24-byte header with magic (defines

codec) variable-length annotation block contiguous segment of audio data

Storage methodology network (big-endian) byte order multi-byte audio data may require byte

reversal in order to operate on it by the arithmetic unit of certain processors

copy DJ Foreman 2012 18

Example 1 Header file for AU files

copy DJ Foreman 2012 19

typedef unsigned long u_32 unsigned 32-bit integer

typedef struct

u_32 magic the ldquomagic numberrdquo

u_32 hdr_size byte offset to start of data

u_32 data_size length (optional)

u_32 encoding data encoding enumeration

u_32 sample_rate samples per second

u_32 channels of interleaved channels

Audio_filehdr

Following is from include ltmultimedialibaudiohgt

AUDIO_FILE_MAGIC ((u_32)0x2e736e64) ldquosndrdquo

Audio Encoding EnumerationsAUDIO_FILE_ENCODING_MULAW_8 (1) 8-bit ISDN u-law

AUDIO_FILE_ENCODING_LINEAR_8 (2) 8-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_16 (3) 16-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_32 (5) 32-bit linear PCM

AUDIO_FILE_ENCODING_FLOAT (6) 32-bit IEEE floating point

AUDIO_FILE_ENCODING_DOUBLE (7) 64-bit IEEE floating point

AUDIO_FILE_ENCODING_ADPCM_G721 (23) 4-bit CCITT g721 ADPCM

AUDIO_FILE_ENCODING_ADPCM_G723_3 (25) CCITT g723 3-bit ADPCM

AUDIO_FILE_ENCODING_ALAW_8 (27) 8-bit ISDN A-law

copy DJ Foreman 2012 20

ldquoLinearrdquo values are SIGNED intrsquosFloats are signed zero-centered normalized to ( -10 lt= x lt= 10 )

Example 2 ZIP files (PkWare)Purpose Size in bytes

Signature header 4

Required version 2

GP flags 2

Method 2

Last mod time 2

Last mod date 2

CRC-32 4

Compressed size 4

Uncompressed size 4

Filename length 2

Extra field length 2

File name variable

extra variable

copy DJ Foreman 2012 21

Ref httplivedocsadobecomflex3htmlhelphtmlcontent=ByteArrays_3html

More info on encoding httpwwwpkwarecomdocumentscase

studiesAPPNOTETXT

httpwwwfileinfocomfiletypescompressed

copy DJ Foreman 2012 22

Recording Bit-Rate

Constant (CBR) rate at which a codecs output data should be

consumed is constant Max bit-rate matters not the average Uses all available bandwidth Not good for storage (lossy)

Variable (VBR) Quantity of datatime unit (for output) varies Better quality for audio and video Slower to encode Supported by most portable devices post 2006

copy DJ Foreman 2012 23

Lab assignment ndash 1 ndash part 1

Write a program with 1 parameter If you are on an encoding team

Open a test file (eg ldquoxyztxtrdquo) Compress it using simple Huffman compression using

the frequency table from my FTP site Output compressed file (eg ldquoxyzencrdquo) to SAME folder

as input

If you are on a decoding team Open compressed file ldquoxyzencrdquo Decompress the input file Output file ldquoxyzdecrdquo so it can be compared to ldquoxyztxtrdquo

The parameter is the full input file path amp name (ie file names MUST NOT be hard-coded)

Remember a code can be a single bit

copy DJ Foreman 2012 24

25

Lab assignment 1 ndash part 2

Use ONLY MY TABLE for the frequencies Use ONLY uppercase letters of the table Do NOT use the special character

tables Find someone in class with the

opposite program and TEST your program to see if it properly encodes or decodes a message

Grading tree construction tree following proper binary output (encoders) proper character output (decoders)

copy DJ Foreman 2012

Working with ImagesModeling

Containers

A container is a FILE format NOT a code scheme

Eg AVI amp WAV are container formats Formats for storage Independent of encoding or content

Others Ogg ASF QuickTime RealMedia Matroska

DivX and MP4

copy DJ Foreman 2012 27

28

Movie (metadata)

Track

Media

Media Information

Container file format (MP4QT)

Chunk map

Comp time

Chunk offset

Key frame

Desc

Decode time

Priority map

Size map

Sample Table

Media Handler

Data information

Media header

Video header

Track header

Movie header

copy DJ Foreman 2012

Pixels A pixel is 3 color-dots (squares) in a collection of dots

(squares) making up the picture Each color-dot is for one of the three primary colors

RGB Minimum of 1 byte per color (depending on bit-depth

which varies with camera) thus at least 3 bytes per pixel

A 10MP camera (30 million color-dots) needs 30M bytes This gets compressed by the camera (JPEG format) Final file size = 30 million bytes divided by the amount

of compression (compression ratio) An 81 compression ratio would reduce that 30 million

bytes to 375 megabytes (as done on a 4 MP camera) A RAW file has NO compression

copy DJ Foreman 2012 29

Image creation

Basic mechanisms Photography

Film Digital ndashPhotoshop Gimp

Drawing Line BrushModeling

Do it yourself OpenGL (2x and 3x)

Modelingrendering programs Breeze designer Ayam 3DCanvas Makehuman Alice

copy DJ Foreman 2012 30

Photography (film) Silver halide crystals (AgCl AgBr AgI)

Exposure to light turns them black Developer removes the loose (exposed) halide leaving

pure silver (a negative image) Fixing bath flushes out unexposed AgHalide

Greatest desire of photographers - a medium with High speed (gathers light quickly) Low noise (no undesired artifacts) High resolution (very detailed images)

Visible lines per mm (lpm) at a specified contrast level

Notes Larger ldquoclumpsrdquo are faster but limit resolution Smaller ldquoclumpsrdquo are hard to coat evenly Randomness of coating prevents Moireacute patternshttpenwikipediaorgwikiMoirC3A9_pattern

copy DJ Foreman 2012 31

FYI ndash a little chemistry

Ag+ X- + hv rarr Ag+ + X + e-

1 Radiation frees the Br2 1 Ag ion + 1 electron (Ag+ + e- rarr Ag0) -gt 1 Ag atom3 Neighbors are now ldquosensitizedrdquo 4 Developing (Reducing) converts exposed AgX to

black metallic Ag and free X5 Remaining AgX and free X are washed away (after

mixing with thiosulfate to make it soluble)

This is one silver (Ag) ldquograinrdquo in the ldquolatent imagerdquo X is any halide (Br Cl etc)

copy DJ Foreman 2012 32

crystal

Radiation (light)

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Overview -1

Required background C C++ or Java CS240 (Data Structures) or CS212 (for

Engineers) This is NOT a graphics course (see

CS460560) Labs

Data compression Elements of graphics in OpenGL Photo manipulation

GIMP Steganography

Creating amp Rendering modelscopy DJ Foreman 2012 2

Topics (general order of coverage)

Compression EncodingDecoding Codecs Huffman encoding

example lab Working with Images

Image creation Photography Drawing RIB (Renderman Interface

Bytestream) Modeling

OpenGL ndash lab Makehuman 3DCanvas etc

lab Image manipulation

Gimp Photoshop etc lab Rendering Bitmaps

Ray tracing

HTML CSS amp Javascript - lab

Animation Hand drawn Program generated

Stop-motion Alice lab DirectX amp OpenGL

Hardware acceleration RIB revisited lab Shaders

Steganography lab Additional topics (time

permitting) Augmented reality Geospatial data systems

copy DJ Foreman 2012 3

Software we might use (all free)

Modelers and environments Ayam ndash ortho amp 3D 3DCanvas ndash 3D Blender ndash ortho amp 3D Makehuman - 3D

Renderers Aqsis Pixie ndash no GUI file input

copy DJ Foreman 2012 4

5

Image formats

Bitmap Also called ldquoraster imagesrdquo

Stored as individual pixels Resolution dependent Screen resolution = 100 ppi Printing requires 150-300 ppi

Vector Scalable primitives amp formulas Resolution independent

copy DJ Foreman 2012

6

Bitmap formats

BMP (program generated) GIF (ldquo but copy limits on use of format) JPEG JPG (photo) PNG (like GIF wo copy limits) PICT(ldquo Mac BMP+Vector) PCX (ldquo Personal Computer Exchange) PSD (Photoshop) TIFF (ldquo can also be a container)

copy DJ Foreman 2012

7

Vector formats

AI (Adobe Illustrator) CDR (CorelDRAW) CMX (Corel Exchange) CGM (Computer Graphics Metafile ) DXF (AutoCAD ) WMF (Windows Metafile )

copy DJ Foreman 2012

8copy DJ Foreman 2012

Compression

Codecs A device or program for encoding andor

decoding a digital data stream or signal Encoding A-gtD for storage or transmission DecodingD-gtA for playback Lossy vs lossless Raw uncompressed Pulse Code Modulation

(PCM) audio a digital representation of an analog signal where the magnitude of the signal is sampled regularly at uniform intervals Eg (441 kHz 16 bit stereo as represented on an

audio CD or in a wav or aiff file) is a standard across multiple platforms

copy DJ Foreman 2012 9

Codecs - 2

May emphasize aspects of data Video

motion vs color Audio

latency (cell phones) vs high-fidelity (music) Data size

transmission speed or storage size vs data loss

copy DJ Foreman 2012 10

Codec References

Forensic Computing A Practitioners Guide by T Sammes amp B Jenkinson (Springer 2000) ISBN 9781852332990

httpwwwgarykesslernetlibraryfile_sigshtml

copy DJ Foreman 2012 11

Considerations

Tradeoffs compression speed compressed data size quality (data loss)

Areas of study information theory rate-distortion theory

Popular algorithms (all lossless) Lempel-Ziv (LZ) LZW (fast decompression) LZR (LZ-Renau) (ZIP files) LZW (Lempel-Ziv-Welch) (for GIF files)

copy DJ Foreman 2012 12

Lossless Encoding

Huffman compression Bit string representing any symbol is never a

prefix of the bit string representing any other symbolie if 010 is the code for a symbol then no other symbol starts with 010

Frequency table must be known computable or included

Lossless ndash every character gets encoded Arithmetic encoding

Probabilistic algorithm Slightly superior to Huffman but often

patentedcopy DJ Foreman 2012 13

Methodology

Compute a model of the data eg a Huffman tree based on probability Probability determined from input file

Map the data according to the model Read 1 symbol Search the model (tree) for that symbol

If itrsquos a Huffman tree going left=0 right=1 Append each 0 or 1 to the code string When symbol is found you are done Note all symbols will have lt 8 bits (thus

compression)

copy DJ Foreman 2012 14

15

Huffman lab ndash more details Read write are always in bytes

You may need to pad You may have extraneous letters in the output

due to padding You donrsquot need to add a header to specify size

You have 2 weeks to do the lab Code may be in C C++ or Java The readme specifies how to compile the

program Encoded data is sent to a binary file or a simple

text file (but isnrsquot ldquoreadablerdquo as text) Encoded data can only be ldquocheckedrdquo with a

decodercopy DJ Foreman 2012

16

Sample of prefix coding amp compression

copy DJ Foreman 2012

B e a r c a t s (8 bytes of text)010101100110001110001111 encoded as123456781234567812345678 3 bytes

A simple Huffman-like code(wo probability basis) 0 1

A 110

B 010

C 0001

E 10

R 011

S 111

T 001

Compression-Model types

Static Read all the data Compute frequencies Re-read the data amp encode

Dynamic Build simple basic model Modify model as more data is processed

copy DJ Foreman 2012 17

Standard Compressed-File Content

Three parts 24-byte header with magic (defines

codec) variable-length annotation block contiguous segment of audio data

Storage methodology network (big-endian) byte order multi-byte audio data may require byte

reversal in order to operate on it by the arithmetic unit of certain processors

copy DJ Foreman 2012 18

Example 1 Header file for AU files

copy DJ Foreman 2012 19

typedef unsigned long u_32 unsigned 32-bit integer

typedef struct

u_32 magic the ldquomagic numberrdquo

u_32 hdr_size byte offset to start of data

u_32 data_size length (optional)

u_32 encoding data encoding enumeration

u_32 sample_rate samples per second

u_32 channels of interleaved channels

Audio_filehdr

Following is from include ltmultimedialibaudiohgt

AUDIO_FILE_MAGIC ((u_32)0x2e736e64) ldquosndrdquo

Audio Encoding EnumerationsAUDIO_FILE_ENCODING_MULAW_8 (1) 8-bit ISDN u-law

AUDIO_FILE_ENCODING_LINEAR_8 (2) 8-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_16 (3) 16-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_32 (5) 32-bit linear PCM

AUDIO_FILE_ENCODING_FLOAT (6) 32-bit IEEE floating point

AUDIO_FILE_ENCODING_DOUBLE (7) 64-bit IEEE floating point

AUDIO_FILE_ENCODING_ADPCM_G721 (23) 4-bit CCITT g721 ADPCM

AUDIO_FILE_ENCODING_ADPCM_G723_3 (25) CCITT g723 3-bit ADPCM

AUDIO_FILE_ENCODING_ALAW_8 (27) 8-bit ISDN A-law

copy DJ Foreman 2012 20

ldquoLinearrdquo values are SIGNED intrsquosFloats are signed zero-centered normalized to ( -10 lt= x lt= 10 )

Example 2 ZIP files (PkWare)Purpose Size in bytes

Signature header 4

Required version 2

GP flags 2

Method 2

Last mod time 2

Last mod date 2

CRC-32 4

Compressed size 4

Uncompressed size 4

Filename length 2

Extra field length 2

File name variable

extra variable

copy DJ Foreman 2012 21

Ref httplivedocsadobecomflex3htmlhelphtmlcontent=ByteArrays_3html

More info on encoding httpwwwpkwarecomdocumentscase

studiesAPPNOTETXT

httpwwwfileinfocomfiletypescompressed

copy DJ Foreman 2012 22

Recording Bit-Rate

Constant (CBR) rate at which a codecs output data should be

consumed is constant Max bit-rate matters not the average Uses all available bandwidth Not good for storage (lossy)

Variable (VBR) Quantity of datatime unit (for output) varies Better quality for audio and video Slower to encode Supported by most portable devices post 2006

copy DJ Foreman 2012 23

Lab assignment ndash 1 ndash part 1

Write a program with 1 parameter If you are on an encoding team

Open a test file (eg ldquoxyztxtrdquo) Compress it using simple Huffman compression using

the frequency table from my FTP site Output compressed file (eg ldquoxyzencrdquo) to SAME folder

as input

If you are on a decoding team Open compressed file ldquoxyzencrdquo Decompress the input file Output file ldquoxyzdecrdquo so it can be compared to ldquoxyztxtrdquo

The parameter is the full input file path amp name (ie file names MUST NOT be hard-coded)

Remember a code can be a single bit

copy DJ Foreman 2012 24

25

Lab assignment 1 ndash part 2

Use ONLY MY TABLE for the frequencies Use ONLY uppercase letters of the table Do NOT use the special character

tables Find someone in class with the

opposite program and TEST your program to see if it properly encodes or decodes a message

Grading tree construction tree following proper binary output (encoders) proper character output (decoders)

copy DJ Foreman 2012

Working with ImagesModeling

Containers

A container is a FILE format NOT a code scheme

Eg AVI amp WAV are container formats Formats for storage Independent of encoding or content

Others Ogg ASF QuickTime RealMedia Matroska

DivX and MP4

copy DJ Foreman 2012 27

28

Movie (metadata)

Track

Media

Media Information

Container file format (MP4QT)

Chunk map

Comp time

Chunk offset

Key frame

Desc

Decode time

Priority map

Size map

Sample Table

Media Handler

Data information

Media header

Video header

Track header

Movie header

copy DJ Foreman 2012

Pixels A pixel is 3 color-dots (squares) in a collection of dots

(squares) making up the picture Each color-dot is for one of the three primary colors

RGB Minimum of 1 byte per color (depending on bit-depth

which varies with camera) thus at least 3 bytes per pixel

A 10MP camera (30 million color-dots) needs 30M bytes This gets compressed by the camera (JPEG format) Final file size = 30 million bytes divided by the amount

of compression (compression ratio) An 81 compression ratio would reduce that 30 million

bytes to 375 megabytes (as done on a 4 MP camera) A RAW file has NO compression

copy DJ Foreman 2012 29

Image creation

Basic mechanisms Photography

Film Digital ndashPhotoshop Gimp

Drawing Line BrushModeling

Do it yourself OpenGL (2x and 3x)

Modelingrendering programs Breeze designer Ayam 3DCanvas Makehuman Alice

copy DJ Foreman 2012 30

Photography (film) Silver halide crystals (AgCl AgBr AgI)

Exposure to light turns them black Developer removes the loose (exposed) halide leaving

pure silver (a negative image) Fixing bath flushes out unexposed AgHalide

Greatest desire of photographers - a medium with High speed (gathers light quickly) Low noise (no undesired artifacts) High resolution (very detailed images)

Visible lines per mm (lpm) at a specified contrast level

Notes Larger ldquoclumpsrdquo are faster but limit resolution Smaller ldquoclumpsrdquo are hard to coat evenly Randomness of coating prevents Moireacute patternshttpenwikipediaorgwikiMoirC3A9_pattern

copy DJ Foreman 2012 31

FYI ndash a little chemistry

Ag+ X- + hv rarr Ag+ + X + e-

1 Radiation frees the Br2 1 Ag ion + 1 electron (Ag+ + e- rarr Ag0) -gt 1 Ag atom3 Neighbors are now ldquosensitizedrdquo 4 Developing (Reducing) converts exposed AgX to

black metallic Ag and free X5 Remaining AgX and free X are washed away (after

mixing with thiosulfate to make it soluble)

This is one silver (Ag) ldquograinrdquo in the ldquolatent imagerdquo X is any halide (Br Cl etc)

copy DJ Foreman 2012 32

crystal

Radiation (light)

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Topics (general order of coverage)

Compression EncodingDecoding Codecs Huffman encoding

example lab Working with Images

Image creation Photography Drawing RIB (Renderman Interface

Bytestream) Modeling

OpenGL ndash lab Makehuman 3DCanvas etc

lab Image manipulation

Gimp Photoshop etc lab Rendering Bitmaps

Ray tracing

HTML CSS amp Javascript - lab

Animation Hand drawn Program generated

Stop-motion Alice lab DirectX amp OpenGL

Hardware acceleration RIB revisited lab Shaders

Steganography lab Additional topics (time

permitting) Augmented reality Geospatial data systems

copy DJ Foreman 2012 3

Software we might use (all free)

Modelers and environments Ayam ndash ortho amp 3D 3DCanvas ndash 3D Blender ndash ortho amp 3D Makehuman - 3D

Renderers Aqsis Pixie ndash no GUI file input

copy DJ Foreman 2012 4

5

Image formats

Bitmap Also called ldquoraster imagesrdquo

Stored as individual pixels Resolution dependent Screen resolution = 100 ppi Printing requires 150-300 ppi

Vector Scalable primitives amp formulas Resolution independent

copy DJ Foreman 2012

6

Bitmap formats

BMP (program generated) GIF (ldquo but copy limits on use of format) JPEG JPG (photo) PNG (like GIF wo copy limits) PICT(ldquo Mac BMP+Vector) PCX (ldquo Personal Computer Exchange) PSD (Photoshop) TIFF (ldquo can also be a container)

copy DJ Foreman 2012

7

Vector formats

AI (Adobe Illustrator) CDR (CorelDRAW) CMX (Corel Exchange) CGM (Computer Graphics Metafile ) DXF (AutoCAD ) WMF (Windows Metafile )

copy DJ Foreman 2012

8copy DJ Foreman 2012

Compression

Codecs A device or program for encoding andor

decoding a digital data stream or signal Encoding A-gtD for storage or transmission DecodingD-gtA for playback Lossy vs lossless Raw uncompressed Pulse Code Modulation

(PCM) audio a digital representation of an analog signal where the magnitude of the signal is sampled regularly at uniform intervals Eg (441 kHz 16 bit stereo as represented on an

audio CD or in a wav or aiff file) is a standard across multiple platforms

copy DJ Foreman 2012 9

Codecs - 2

May emphasize aspects of data Video

motion vs color Audio

latency (cell phones) vs high-fidelity (music) Data size

transmission speed or storage size vs data loss

copy DJ Foreman 2012 10

Codec References

Forensic Computing A Practitioners Guide by T Sammes amp B Jenkinson (Springer 2000) ISBN 9781852332990

httpwwwgarykesslernetlibraryfile_sigshtml

copy DJ Foreman 2012 11

Considerations

Tradeoffs compression speed compressed data size quality (data loss)

Areas of study information theory rate-distortion theory

Popular algorithms (all lossless) Lempel-Ziv (LZ) LZW (fast decompression) LZR (LZ-Renau) (ZIP files) LZW (Lempel-Ziv-Welch) (for GIF files)

copy DJ Foreman 2012 12

Lossless Encoding

Huffman compression Bit string representing any symbol is never a

prefix of the bit string representing any other symbolie if 010 is the code for a symbol then no other symbol starts with 010

Frequency table must be known computable or included

Lossless ndash every character gets encoded Arithmetic encoding

Probabilistic algorithm Slightly superior to Huffman but often

patentedcopy DJ Foreman 2012 13

Methodology

Compute a model of the data eg a Huffman tree based on probability Probability determined from input file

Map the data according to the model Read 1 symbol Search the model (tree) for that symbol

If itrsquos a Huffman tree going left=0 right=1 Append each 0 or 1 to the code string When symbol is found you are done Note all symbols will have lt 8 bits (thus

compression)

copy DJ Foreman 2012 14

15

Huffman lab ndash more details Read write are always in bytes

You may need to pad You may have extraneous letters in the output

due to padding You donrsquot need to add a header to specify size

You have 2 weeks to do the lab Code may be in C C++ or Java The readme specifies how to compile the

program Encoded data is sent to a binary file or a simple

text file (but isnrsquot ldquoreadablerdquo as text) Encoded data can only be ldquocheckedrdquo with a

decodercopy DJ Foreman 2012

16

Sample of prefix coding amp compression

copy DJ Foreman 2012

B e a r c a t s (8 bytes of text)010101100110001110001111 encoded as123456781234567812345678 3 bytes

A simple Huffman-like code(wo probability basis) 0 1

A 110

B 010

C 0001

E 10

R 011

S 111

T 001

Compression-Model types

Static Read all the data Compute frequencies Re-read the data amp encode

Dynamic Build simple basic model Modify model as more data is processed

copy DJ Foreman 2012 17

Standard Compressed-File Content

Three parts 24-byte header with magic (defines

codec) variable-length annotation block contiguous segment of audio data

Storage methodology network (big-endian) byte order multi-byte audio data may require byte

reversal in order to operate on it by the arithmetic unit of certain processors

copy DJ Foreman 2012 18

Example 1 Header file for AU files

copy DJ Foreman 2012 19

typedef unsigned long u_32 unsigned 32-bit integer

typedef struct

u_32 magic the ldquomagic numberrdquo

u_32 hdr_size byte offset to start of data

u_32 data_size length (optional)

u_32 encoding data encoding enumeration

u_32 sample_rate samples per second

u_32 channels of interleaved channels

Audio_filehdr

Following is from include ltmultimedialibaudiohgt

AUDIO_FILE_MAGIC ((u_32)0x2e736e64) ldquosndrdquo

Audio Encoding EnumerationsAUDIO_FILE_ENCODING_MULAW_8 (1) 8-bit ISDN u-law

AUDIO_FILE_ENCODING_LINEAR_8 (2) 8-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_16 (3) 16-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_32 (5) 32-bit linear PCM

AUDIO_FILE_ENCODING_FLOAT (6) 32-bit IEEE floating point

AUDIO_FILE_ENCODING_DOUBLE (7) 64-bit IEEE floating point

AUDIO_FILE_ENCODING_ADPCM_G721 (23) 4-bit CCITT g721 ADPCM

AUDIO_FILE_ENCODING_ADPCM_G723_3 (25) CCITT g723 3-bit ADPCM

AUDIO_FILE_ENCODING_ALAW_8 (27) 8-bit ISDN A-law

copy DJ Foreman 2012 20

ldquoLinearrdquo values are SIGNED intrsquosFloats are signed zero-centered normalized to ( -10 lt= x lt= 10 )

Example 2 ZIP files (PkWare)Purpose Size in bytes

Signature header 4

Required version 2

GP flags 2

Method 2

Last mod time 2

Last mod date 2

CRC-32 4

Compressed size 4

Uncompressed size 4

Filename length 2

Extra field length 2

File name variable

extra variable

copy DJ Foreman 2012 21

Ref httplivedocsadobecomflex3htmlhelphtmlcontent=ByteArrays_3html

More info on encoding httpwwwpkwarecomdocumentscase

studiesAPPNOTETXT

httpwwwfileinfocomfiletypescompressed

copy DJ Foreman 2012 22

Recording Bit-Rate

Constant (CBR) rate at which a codecs output data should be

consumed is constant Max bit-rate matters not the average Uses all available bandwidth Not good for storage (lossy)

Variable (VBR) Quantity of datatime unit (for output) varies Better quality for audio and video Slower to encode Supported by most portable devices post 2006

copy DJ Foreman 2012 23

Lab assignment ndash 1 ndash part 1

Write a program with 1 parameter If you are on an encoding team

Open a test file (eg ldquoxyztxtrdquo) Compress it using simple Huffman compression using

the frequency table from my FTP site Output compressed file (eg ldquoxyzencrdquo) to SAME folder

as input

If you are on a decoding team Open compressed file ldquoxyzencrdquo Decompress the input file Output file ldquoxyzdecrdquo so it can be compared to ldquoxyztxtrdquo

The parameter is the full input file path amp name (ie file names MUST NOT be hard-coded)

Remember a code can be a single bit

copy DJ Foreman 2012 24

25

Lab assignment 1 ndash part 2

Use ONLY MY TABLE for the frequencies Use ONLY uppercase letters of the table Do NOT use the special character

tables Find someone in class with the

opposite program and TEST your program to see if it properly encodes or decodes a message

Grading tree construction tree following proper binary output (encoders) proper character output (decoders)

copy DJ Foreman 2012

Working with ImagesModeling

Containers

A container is a FILE format NOT a code scheme

Eg AVI amp WAV are container formats Formats for storage Independent of encoding or content

Others Ogg ASF QuickTime RealMedia Matroska

DivX and MP4

copy DJ Foreman 2012 27

28

Movie (metadata)

Track

Media

Media Information

Container file format (MP4QT)

Chunk map

Comp time

Chunk offset

Key frame

Desc

Decode time

Priority map

Size map

Sample Table

Media Handler

Data information

Media header

Video header

Track header

Movie header

copy DJ Foreman 2012

Pixels A pixel is 3 color-dots (squares) in a collection of dots

(squares) making up the picture Each color-dot is for one of the three primary colors

RGB Minimum of 1 byte per color (depending on bit-depth

which varies with camera) thus at least 3 bytes per pixel

A 10MP camera (30 million color-dots) needs 30M bytes This gets compressed by the camera (JPEG format) Final file size = 30 million bytes divided by the amount

of compression (compression ratio) An 81 compression ratio would reduce that 30 million

bytes to 375 megabytes (as done on a 4 MP camera) A RAW file has NO compression

copy DJ Foreman 2012 29

Image creation

Basic mechanisms Photography

Film Digital ndashPhotoshop Gimp

Drawing Line BrushModeling

Do it yourself OpenGL (2x and 3x)

Modelingrendering programs Breeze designer Ayam 3DCanvas Makehuman Alice

copy DJ Foreman 2012 30

Photography (film) Silver halide crystals (AgCl AgBr AgI)

Exposure to light turns them black Developer removes the loose (exposed) halide leaving

pure silver (a negative image) Fixing bath flushes out unexposed AgHalide

Greatest desire of photographers - a medium with High speed (gathers light quickly) Low noise (no undesired artifacts) High resolution (very detailed images)

Visible lines per mm (lpm) at a specified contrast level

Notes Larger ldquoclumpsrdquo are faster but limit resolution Smaller ldquoclumpsrdquo are hard to coat evenly Randomness of coating prevents Moireacute patternshttpenwikipediaorgwikiMoirC3A9_pattern

copy DJ Foreman 2012 31

FYI ndash a little chemistry

Ag+ X- + hv rarr Ag+ + X + e-

1 Radiation frees the Br2 1 Ag ion + 1 electron (Ag+ + e- rarr Ag0) -gt 1 Ag atom3 Neighbors are now ldquosensitizedrdquo 4 Developing (Reducing) converts exposed AgX to

black metallic Ag and free X5 Remaining AgX and free X are washed away (after

mixing with thiosulfate to make it soluble)

This is one silver (Ag) ldquograinrdquo in the ldquolatent imagerdquo X is any halide (Br Cl etc)

copy DJ Foreman 2012 32

crystal

Radiation (light)

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Software we might use (all free)

Modelers and environments Ayam ndash ortho amp 3D 3DCanvas ndash 3D Blender ndash ortho amp 3D Makehuman - 3D

Renderers Aqsis Pixie ndash no GUI file input

copy DJ Foreman 2012 4

5

Image formats

Bitmap Also called ldquoraster imagesrdquo

Stored as individual pixels Resolution dependent Screen resolution = 100 ppi Printing requires 150-300 ppi

Vector Scalable primitives amp formulas Resolution independent

copy DJ Foreman 2012

6

Bitmap formats

BMP (program generated) GIF (ldquo but copy limits on use of format) JPEG JPG (photo) PNG (like GIF wo copy limits) PICT(ldquo Mac BMP+Vector) PCX (ldquo Personal Computer Exchange) PSD (Photoshop) TIFF (ldquo can also be a container)

copy DJ Foreman 2012

7

Vector formats

AI (Adobe Illustrator) CDR (CorelDRAW) CMX (Corel Exchange) CGM (Computer Graphics Metafile ) DXF (AutoCAD ) WMF (Windows Metafile )

copy DJ Foreman 2012

8copy DJ Foreman 2012

Compression

Codecs A device or program for encoding andor

decoding a digital data stream or signal Encoding A-gtD for storage or transmission DecodingD-gtA for playback Lossy vs lossless Raw uncompressed Pulse Code Modulation

(PCM) audio a digital representation of an analog signal where the magnitude of the signal is sampled regularly at uniform intervals Eg (441 kHz 16 bit stereo as represented on an

audio CD or in a wav or aiff file) is a standard across multiple platforms

copy DJ Foreman 2012 9

Codecs - 2

May emphasize aspects of data Video

motion vs color Audio

latency (cell phones) vs high-fidelity (music) Data size

transmission speed or storage size vs data loss

copy DJ Foreman 2012 10

Codec References

Forensic Computing A Practitioners Guide by T Sammes amp B Jenkinson (Springer 2000) ISBN 9781852332990

httpwwwgarykesslernetlibraryfile_sigshtml

copy DJ Foreman 2012 11

Considerations

Tradeoffs compression speed compressed data size quality (data loss)

Areas of study information theory rate-distortion theory

Popular algorithms (all lossless) Lempel-Ziv (LZ) LZW (fast decompression) LZR (LZ-Renau) (ZIP files) LZW (Lempel-Ziv-Welch) (for GIF files)

copy DJ Foreman 2012 12

Lossless Encoding

Huffman compression Bit string representing any symbol is never a

prefix of the bit string representing any other symbolie if 010 is the code for a symbol then no other symbol starts with 010

Frequency table must be known computable or included

Lossless ndash every character gets encoded Arithmetic encoding

Probabilistic algorithm Slightly superior to Huffman but often

patentedcopy DJ Foreman 2012 13

Methodology

Compute a model of the data eg a Huffman tree based on probability Probability determined from input file

Map the data according to the model Read 1 symbol Search the model (tree) for that symbol

If itrsquos a Huffman tree going left=0 right=1 Append each 0 or 1 to the code string When symbol is found you are done Note all symbols will have lt 8 bits (thus

compression)

copy DJ Foreman 2012 14

15

Huffman lab ndash more details Read write are always in bytes

You may need to pad You may have extraneous letters in the output

due to padding You donrsquot need to add a header to specify size

You have 2 weeks to do the lab Code may be in C C++ or Java The readme specifies how to compile the

program Encoded data is sent to a binary file or a simple

text file (but isnrsquot ldquoreadablerdquo as text) Encoded data can only be ldquocheckedrdquo with a

decodercopy DJ Foreman 2012

16

Sample of prefix coding amp compression

copy DJ Foreman 2012

B e a r c a t s (8 bytes of text)010101100110001110001111 encoded as123456781234567812345678 3 bytes

A simple Huffman-like code(wo probability basis) 0 1

A 110

B 010

C 0001

E 10

R 011

S 111

T 001

Compression-Model types

Static Read all the data Compute frequencies Re-read the data amp encode

Dynamic Build simple basic model Modify model as more data is processed

copy DJ Foreman 2012 17

Standard Compressed-File Content

Three parts 24-byte header with magic (defines

codec) variable-length annotation block contiguous segment of audio data

Storage methodology network (big-endian) byte order multi-byte audio data may require byte

reversal in order to operate on it by the arithmetic unit of certain processors

copy DJ Foreman 2012 18

Example 1 Header file for AU files

copy DJ Foreman 2012 19

typedef unsigned long u_32 unsigned 32-bit integer

typedef struct

u_32 magic the ldquomagic numberrdquo

u_32 hdr_size byte offset to start of data

u_32 data_size length (optional)

u_32 encoding data encoding enumeration

u_32 sample_rate samples per second

u_32 channels of interleaved channels

Audio_filehdr

Following is from include ltmultimedialibaudiohgt

AUDIO_FILE_MAGIC ((u_32)0x2e736e64) ldquosndrdquo

Audio Encoding EnumerationsAUDIO_FILE_ENCODING_MULAW_8 (1) 8-bit ISDN u-law

AUDIO_FILE_ENCODING_LINEAR_8 (2) 8-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_16 (3) 16-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_32 (5) 32-bit linear PCM

AUDIO_FILE_ENCODING_FLOAT (6) 32-bit IEEE floating point

AUDIO_FILE_ENCODING_DOUBLE (7) 64-bit IEEE floating point

AUDIO_FILE_ENCODING_ADPCM_G721 (23) 4-bit CCITT g721 ADPCM

AUDIO_FILE_ENCODING_ADPCM_G723_3 (25) CCITT g723 3-bit ADPCM

AUDIO_FILE_ENCODING_ALAW_8 (27) 8-bit ISDN A-law

copy DJ Foreman 2012 20

ldquoLinearrdquo values are SIGNED intrsquosFloats are signed zero-centered normalized to ( -10 lt= x lt= 10 )

Example 2 ZIP files (PkWare)Purpose Size in bytes

Signature header 4

Required version 2

GP flags 2

Method 2

Last mod time 2

Last mod date 2

CRC-32 4

Compressed size 4

Uncompressed size 4

Filename length 2

Extra field length 2

File name variable

extra variable

copy DJ Foreman 2012 21

Ref httplivedocsadobecomflex3htmlhelphtmlcontent=ByteArrays_3html

More info on encoding httpwwwpkwarecomdocumentscase

studiesAPPNOTETXT

httpwwwfileinfocomfiletypescompressed

copy DJ Foreman 2012 22

Recording Bit-Rate

Constant (CBR) rate at which a codecs output data should be

consumed is constant Max bit-rate matters not the average Uses all available bandwidth Not good for storage (lossy)

Variable (VBR) Quantity of datatime unit (for output) varies Better quality for audio and video Slower to encode Supported by most portable devices post 2006

copy DJ Foreman 2012 23

Lab assignment ndash 1 ndash part 1

Write a program with 1 parameter If you are on an encoding team

Open a test file (eg ldquoxyztxtrdquo) Compress it using simple Huffman compression using

the frequency table from my FTP site Output compressed file (eg ldquoxyzencrdquo) to SAME folder

as input

If you are on a decoding team Open compressed file ldquoxyzencrdquo Decompress the input file Output file ldquoxyzdecrdquo so it can be compared to ldquoxyztxtrdquo

The parameter is the full input file path amp name (ie file names MUST NOT be hard-coded)

Remember a code can be a single bit

copy DJ Foreman 2012 24

25

Lab assignment 1 ndash part 2

Use ONLY MY TABLE for the frequencies Use ONLY uppercase letters of the table Do NOT use the special character

tables Find someone in class with the

opposite program and TEST your program to see if it properly encodes or decodes a message

Grading tree construction tree following proper binary output (encoders) proper character output (decoders)

copy DJ Foreman 2012

Working with ImagesModeling

Containers

A container is a FILE format NOT a code scheme

Eg AVI amp WAV are container formats Formats for storage Independent of encoding or content

Others Ogg ASF QuickTime RealMedia Matroska

DivX and MP4

copy DJ Foreman 2012 27

28

Movie (metadata)

Track

Media

Media Information

Container file format (MP4QT)

Chunk map

Comp time

Chunk offset

Key frame

Desc

Decode time

Priority map

Size map

Sample Table

Media Handler

Data information

Media header

Video header

Track header

Movie header

copy DJ Foreman 2012

Pixels A pixel is 3 color-dots (squares) in a collection of dots

(squares) making up the picture Each color-dot is for one of the three primary colors

RGB Minimum of 1 byte per color (depending on bit-depth

which varies with camera) thus at least 3 bytes per pixel

A 10MP camera (30 million color-dots) needs 30M bytes This gets compressed by the camera (JPEG format) Final file size = 30 million bytes divided by the amount

of compression (compression ratio) An 81 compression ratio would reduce that 30 million

bytes to 375 megabytes (as done on a 4 MP camera) A RAW file has NO compression

copy DJ Foreman 2012 29

Image creation

Basic mechanisms Photography

Film Digital ndashPhotoshop Gimp

Drawing Line BrushModeling

Do it yourself OpenGL (2x and 3x)

Modelingrendering programs Breeze designer Ayam 3DCanvas Makehuman Alice

copy DJ Foreman 2012 30

Photography (film) Silver halide crystals (AgCl AgBr AgI)

Exposure to light turns them black Developer removes the loose (exposed) halide leaving

pure silver (a negative image) Fixing bath flushes out unexposed AgHalide

Greatest desire of photographers - a medium with High speed (gathers light quickly) Low noise (no undesired artifacts) High resolution (very detailed images)

Visible lines per mm (lpm) at a specified contrast level

Notes Larger ldquoclumpsrdquo are faster but limit resolution Smaller ldquoclumpsrdquo are hard to coat evenly Randomness of coating prevents Moireacute patternshttpenwikipediaorgwikiMoirC3A9_pattern

copy DJ Foreman 2012 31

FYI ndash a little chemistry

Ag+ X- + hv rarr Ag+ + X + e-

1 Radiation frees the Br2 1 Ag ion + 1 electron (Ag+ + e- rarr Ag0) -gt 1 Ag atom3 Neighbors are now ldquosensitizedrdquo 4 Developing (Reducing) converts exposed AgX to

black metallic Ag and free X5 Remaining AgX and free X are washed away (after

mixing with thiosulfate to make it soluble)

This is one silver (Ag) ldquograinrdquo in the ldquolatent imagerdquo X is any halide (Br Cl etc)

copy DJ Foreman 2012 32

crystal

Radiation (light)

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

5

Image formats

Bitmap Also called ldquoraster imagesrdquo

Stored as individual pixels Resolution dependent Screen resolution = 100 ppi Printing requires 150-300 ppi

Vector Scalable primitives amp formulas Resolution independent

copy DJ Foreman 2012

6

Bitmap formats

BMP (program generated) GIF (ldquo but copy limits on use of format) JPEG JPG (photo) PNG (like GIF wo copy limits) PICT(ldquo Mac BMP+Vector) PCX (ldquo Personal Computer Exchange) PSD (Photoshop) TIFF (ldquo can also be a container)

copy DJ Foreman 2012

7

Vector formats

AI (Adobe Illustrator) CDR (CorelDRAW) CMX (Corel Exchange) CGM (Computer Graphics Metafile ) DXF (AutoCAD ) WMF (Windows Metafile )

copy DJ Foreman 2012

8copy DJ Foreman 2012

Compression

Codecs A device or program for encoding andor

decoding a digital data stream or signal Encoding A-gtD for storage or transmission DecodingD-gtA for playback Lossy vs lossless Raw uncompressed Pulse Code Modulation

(PCM) audio a digital representation of an analog signal where the magnitude of the signal is sampled regularly at uniform intervals Eg (441 kHz 16 bit stereo as represented on an

audio CD or in a wav or aiff file) is a standard across multiple platforms

copy DJ Foreman 2012 9

Codecs - 2

May emphasize aspects of data Video

motion vs color Audio

latency (cell phones) vs high-fidelity (music) Data size

transmission speed or storage size vs data loss

copy DJ Foreman 2012 10

Codec References

Forensic Computing A Practitioners Guide by T Sammes amp B Jenkinson (Springer 2000) ISBN 9781852332990

httpwwwgarykesslernetlibraryfile_sigshtml

copy DJ Foreman 2012 11

Considerations

Tradeoffs compression speed compressed data size quality (data loss)

Areas of study information theory rate-distortion theory

Popular algorithms (all lossless) Lempel-Ziv (LZ) LZW (fast decompression) LZR (LZ-Renau) (ZIP files) LZW (Lempel-Ziv-Welch) (for GIF files)

copy DJ Foreman 2012 12

Lossless Encoding

Huffman compression Bit string representing any symbol is never a

prefix of the bit string representing any other symbolie if 010 is the code for a symbol then no other symbol starts with 010

Frequency table must be known computable or included

Lossless ndash every character gets encoded Arithmetic encoding

Probabilistic algorithm Slightly superior to Huffman but often

patentedcopy DJ Foreman 2012 13

Methodology

Compute a model of the data eg a Huffman tree based on probability Probability determined from input file

Map the data according to the model Read 1 symbol Search the model (tree) for that symbol

If itrsquos a Huffman tree going left=0 right=1 Append each 0 or 1 to the code string When symbol is found you are done Note all symbols will have lt 8 bits (thus

compression)

copy DJ Foreman 2012 14

15

Huffman lab ndash more details Read write are always in bytes

You may need to pad You may have extraneous letters in the output

due to padding You donrsquot need to add a header to specify size

You have 2 weeks to do the lab Code may be in C C++ or Java The readme specifies how to compile the

program Encoded data is sent to a binary file or a simple

text file (but isnrsquot ldquoreadablerdquo as text) Encoded data can only be ldquocheckedrdquo with a

decodercopy DJ Foreman 2012

16

Sample of prefix coding amp compression

copy DJ Foreman 2012

B e a r c a t s (8 bytes of text)010101100110001110001111 encoded as123456781234567812345678 3 bytes

A simple Huffman-like code(wo probability basis) 0 1

A 110

B 010

C 0001

E 10

R 011

S 111

T 001

Compression-Model types

Static Read all the data Compute frequencies Re-read the data amp encode

Dynamic Build simple basic model Modify model as more data is processed

copy DJ Foreman 2012 17

Standard Compressed-File Content

Three parts 24-byte header with magic (defines

codec) variable-length annotation block contiguous segment of audio data

Storage methodology network (big-endian) byte order multi-byte audio data may require byte

reversal in order to operate on it by the arithmetic unit of certain processors

copy DJ Foreman 2012 18

Example 1 Header file for AU files

copy DJ Foreman 2012 19

typedef unsigned long u_32 unsigned 32-bit integer

typedef struct

u_32 magic the ldquomagic numberrdquo

u_32 hdr_size byte offset to start of data

u_32 data_size length (optional)

u_32 encoding data encoding enumeration

u_32 sample_rate samples per second

u_32 channels of interleaved channels

Audio_filehdr

Following is from include ltmultimedialibaudiohgt

AUDIO_FILE_MAGIC ((u_32)0x2e736e64) ldquosndrdquo

Audio Encoding EnumerationsAUDIO_FILE_ENCODING_MULAW_8 (1) 8-bit ISDN u-law

AUDIO_FILE_ENCODING_LINEAR_8 (2) 8-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_16 (3) 16-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_32 (5) 32-bit linear PCM

AUDIO_FILE_ENCODING_FLOAT (6) 32-bit IEEE floating point

AUDIO_FILE_ENCODING_DOUBLE (7) 64-bit IEEE floating point

AUDIO_FILE_ENCODING_ADPCM_G721 (23) 4-bit CCITT g721 ADPCM

AUDIO_FILE_ENCODING_ADPCM_G723_3 (25) CCITT g723 3-bit ADPCM

AUDIO_FILE_ENCODING_ALAW_8 (27) 8-bit ISDN A-law

copy DJ Foreman 2012 20

ldquoLinearrdquo values are SIGNED intrsquosFloats are signed zero-centered normalized to ( -10 lt= x lt= 10 )

Example 2 ZIP files (PkWare)Purpose Size in bytes

Signature header 4

Required version 2

GP flags 2

Method 2

Last mod time 2

Last mod date 2

CRC-32 4

Compressed size 4

Uncompressed size 4

Filename length 2

Extra field length 2

File name variable

extra variable

copy DJ Foreman 2012 21

Ref httplivedocsadobecomflex3htmlhelphtmlcontent=ByteArrays_3html

More info on encoding httpwwwpkwarecomdocumentscase

studiesAPPNOTETXT

httpwwwfileinfocomfiletypescompressed

copy DJ Foreman 2012 22

Recording Bit-Rate

Constant (CBR) rate at which a codecs output data should be

consumed is constant Max bit-rate matters not the average Uses all available bandwidth Not good for storage (lossy)

Variable (VBR) Quantity of datatime unit (for output) varies Better quality for audio and video Slower to encode Supported by most portable devices post 2006

copy DJ Foreman 2012 23

Lab assignment ndash 1 ndash part 1

Write a program with 1 parameter If you are on an encoding team

Open a test file (eg ldquoxyztxtrdquo) Compress it using simple Huffman compression using

the frequency table from my FTP site Output compressed file (eg ldquoxyzencrdquo) to SAME folder

as input

If you are on a decoding team Open compressed file ldquoxyzencrdquo Decompress the input file Output file ldquoxyzdecrdquo so it can be compared to ldquoxyztxtrdquo

The parameter is the full input file path amp name (ie file names MUST NOT be hard-coded)

Remember a code can be a single bit

copy DJ Foreman 2012 24

25

Lab assignment 1 ndash part 2

Use ONLY MY TABLE for the frequencies Use ONLY uppercase letters of the table Do NOT use the special character

tables Find someone in class with the

opposite program and TEST your program to see if it properly encodes or decodes a message

Grading tree construction tree following proper binary output (encoders) proper character output (decoders)

copy DJ Foreman 2012

Working with ImagesModeling

Containers

A container is a FILE format NOT a code scheme

Eg AVI amp WAV are container formats Formats for storage Independent of encoding or content

Others Ogg ASF QuickTime RealMedia Matroska

DivX and MP4

copy DJ Foreman 2012 27

28

Movie (metadata)

Track

Media

Media Information

Container file format (MP4QT)

Chunk map

Comp time

Chunk offset

Key frame

Desc

Decode time

Priority map

Size map

Sample Table

Media Handler

Data information

Media header

Video header

Track header

Movie header

copy DJ Foreman 2012

Pixels A pixel is 3 color-dots (squares) in a collection of dots

(squares) making up the picture Each color-dot is for one of the three primary colors

RGB Minimum of 1 byte per color (depending on bit-depth

which varies with camera) thus at least 3 bytes per pixel

A 10MP camera (30 million color-dots) needs 30M bytes This gets compressed by the camera (JPEG format) Final file size = 30 million bytes divided by the amount

of compression (compression ratio) An 81 compression ratio would reduce that 30 million

bytes to 375 megabytes (as done on a 4 MP camera) A RAW file has NO compression

copy DJ Foreman 2012 29

Image creation

Basic mechanisms Photography

Film Digital ndashPhotoshop Gimp

Drawing Line BrushModeling

Do it yourself OpenGL (2x and 3x)

Modelingrendering programs Breeze designer Ayam 3DCanvas Makehuman Alice

copy DJ Foreman 2012 30

Photography (film) Silver halide crystals (AgCl AgBr AgI)

Exposure to light turns them black Developer removes the loose (exposed) halide leaving

pure silver (a negative image) Fixing bath flushes out unexposed AgHalide

Greatest desire of photographers - a medium with High speed (gathers light quickly) Low noise (no undesired artifacts) High resolution (very detailed images)

Visible lines per mm (lpm) at a specified contrast level

Notes Larger ldquoclumpsrdquo are faster but limit resolution Smaller ldquoclumpsrdquo are hard to coat evenly Randomness of coating prevents Moireacute patternshttpenwikipediaorgwikiMoirC3A9_pattern

copy DJ Foreman 2012 31

FYI ndash a little chemistry

Ag+ X- + hv rarr Ag+ + X + e-

1 Radiation frees the Br2 1 Ag ion + 1 electron (Ag+ + e- rarr Ag0) -gt 1 Ag atom3 Neighbors are now ldquosensitizedrdquo 4 Developing (Reducing) converts exposed AgX to

black metallic Ag and free X5 Remaining AgX and free X are washed away (after

mixing with thiosulfate to make it soluble)

This is one silver (Ag) ldquograinrdquo in the ldquolatent imagerdquo X is any halide (Br Cl etc)

copy DJ Foreman 2012 32

crystal

Radiation (light)

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

6

Bitmap formats

BMP (program generated) GIF (ldquo but copy limits on use of format) JPEG JPG (photo) PNG (like GIF wo copy limits) PICT(ldquo Mac BMP+Vector) PCX (ldquo Personal Computer Exchange) PSD (Photoshop) TIFF (ldquo can also be a container)

copy DJ Foreman 2012

7

Vector formats

AI (Adobe Illustrator) CDR (CorelDRAW) CMX (Corel Exchange) CGM (Computer Graphics Metafile ) DXF (AutoCAD ) WMF (Windows Metafile )

copy DJ Foreman 2012

8copy DJ Foreman 2012

Compression

Codecs A device or program for encoding andor

decoding a digital data stream or signal Encoding A-gtD for storage or transmission DecodingD-gtA for playback Lossy vs lossless Raw uncompressed Pulse Code Modulation

(PCM) audio a digital representation of an analog signal where the magnitude of the signal is sampled regularly at uniform intervals Eg (441 kHz 16 bit stereo as represented on an

audio CD or in a wav or aiff file) is a standard across multiple platforms

copy DJ Foreman 2012 9

Codecs - 2

May emphasize aspects of data Video

motion vs color Audio

latency (cell phones) vs high-fidelity (music) Data size

transmission speed or storage size vs data loss

copy DJ Foreman 2012 10

Codec References

Forensic Computing A Practitioners Guide by T Sammes amp B Jenkinson (Springer 2000) ISBN 9781852332990

httpwwwgarykesslernetlibraryfile_sigshtml

copy DJ Foreman 2012 11

Considerations

Tradeoffs compression speed compressed data size quality (data loss)

Areas of study information theory rate-distortion theory

Popular algorithms (all lossless) Lempel-Ziv (LZ) LZW (fast decompression) LZR (LZ-Renau) (ZIP files) LZW (Lempel-Ziv-Welch) (for GIF files)

copy DJ Foreman 2012 12

Lossless Encoding

Huffman compression Bit string representing any symbol is never a

prefix of the bit string representing any other symbolie if 010 is the code for a symbol then no other symbol starts with 010

Frequency table must be known computable or included

Lossless ndash every character gets encoded Arithmetic encoding

Probabilistic algorithm Slightly superior to Huffman but often

patentedcopy DJ Foreman 2012 13

Methodology

Compute a model of the data eg a Huffman tree based on probability Probability determined from input file

Map the data according to the model Read 1 symbol Search the model (tree) for that symbol

If itrsquos a Huffman tree going left=0 right=1 Append each 0 or 1 to the code string When symbol is found you are done Note all symbols will have lt 8 bits (thus

compression)

copy DJ Foreman 2012 14

15

Huffman lab ndash more details Read write are always in bytes

You may need to pad You may have extraneous letters in the output

due to padding You donrsquot need to add a header to specify size

You have 2 weeks to do the lab Code may be in C C++ or Java The readme specifies how to compile the

program Encoded data is sent to a binary file or a simple

text file (but isnrsquot ldquoreadablerdquo as text) Encoded data can only be ldquocheckedrdquo with a

decodercopy DJ Foreman 2012

16

Sample of prefix coding amp compression

copy DJ Foreman 2012

B e a r c a t s (8 bytes of text)010101100110001110001111 encoded as123456781234567812345678 3 bytes

A simple Huffman-like code(wo probability basis) 0 1

A 110

B 010

C 0001

E 10

R 011

S 111

T 001

Compression-Model types

Static Read all the data Compute frequencies Re-read the data amp encode

Dynamic Build simple basic model Modify model as more data is processed

copy DJ Foreman 2012 17

Standard Compressed-File Content

Three parts 24-byte header with magic (defines

codec) variable-length annotation block contiguous segment of audio data

Storage methodology network (big-endian) byte order multi-byte audio data may require byte

reversal in order to operate on it by the arithmetic unit of certain processors

copy DJ Foreman 2012 18

Example 1 Header file for AU files

copy DJ Foreman 2012 19

typedef unsigned long u_32 unsigned 32-bit integer

typedef struct

u_32 magic the ldquomagic numberrdquo

u_32 hdr_size byte offset to start of data

u_32 data_size length (optional)

u_32 encoding data encoding enumeration

u_32 sample_rate samples per second

u_32 channels of interleaved channels

Audio_filehdr

Following is from include ltmultimedialibaudiohgt

AUDIO_FILE_MAGIC ((u_32)0x2e736e64) ldquosndrdquo

Audio Encoding EnumerationsAUDIO_FILE_ENCODING_MULAW_8 (1) 8-bit ISDN u-law

AUDIO_FILE_ENCODING_LINEAR_8 (2) 8-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_16 (3) 16-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_32 (5) 32-bit linear PCM

AUDIO_FILE_ENCODING_FLOAT (6) 32-bit IEEE floating point

AUDIO_FILE_ENCODING_DOUBLE (7) 64-bit IEEE floating point

AUDIO_FILE_ENCODING_ADPCM_G721 (23) 4-bit CCITT g721 ADPCM

AUDIO_FILE_ENCODING_ADPCM_G723_3 (25) CCITT g723 3-bit ADPCM

AUDIO_FILE_ENCODING_ALAW_8 (27) 8-bit ISDN A-law

copy DJ Foreman 2012 20

ldquoLinearrdquo values are SIGNED intrsquosFloats are signed zero-centered normalized to ( -10 lt= x lt= 10 )

Example 2 ZIP files (PkWare)Purpose Size in bytes

Signature header 4

Required version 2

GP flags 2

Method 2

Last mod time 2

Last mod date 2

CRC-32 4

Compressed size 4

Uncompressed size 4

Filename length 2

Extra field length 2

File name variable

extra variable

copy DJ Foreman 2012 21

Ref httplivedocsadobecomflex3htmlhelphtmlcontent=ByteArrays_3html

More info on encoding httpwwwpkwarecomdocumentscase

studiesAPPNOTETXT

httpwwwfileinfocomfiletypescompressed

copy DJ Foreman 2012 22

Recording Bit-Rate

Constant (CBR) rate at which a codecs output data should be

consumed is constant Max bit-rate matters not the average Uses all available bandwidth Not good for storage (lossy)

Variable (VBR) Quantity of datatime unit (for output) varies Better quality for audio and video Slower to encode Supported by most portable devices post 2006

copy DJ Foreman 2012 23

Lab assignment ndash 1 ndash part 1

Write a program with 1 parameter If you are on an encoding team

Open a test file (eg ldquoxyztxtrdquo) Compress it using simple Huffman compression using

the frequency table from my FTP site Output compressed file (eg ldquoxyzencrdquo) to SAME folder

as input

If you are on a decoding team Open compressed file ldquoxyzencrdquo Decompress the input file Output file ldquoxyzdecrdquo so it can be compared to ldquoxyztxtrdquo

The parameter is the full input file path amp name (ie file names MUST NOT be hard-coded)

Remember a code can be a single bit

copy DJ Foreman 2012 24

25

Lab assignment 1 ndash part 2

Use ONLY MY TABLE for the frequencies Use ONLY uppercase letters of the table Do NOT use the special character

tables Find someone in class with the

opposite program and TEST your program to see if it properly encodes or decodes a message

Grading tree construction tree following proper binary output (encoders) proper character output (decoders)

copy DJ Foreman 2012

Working with ImagesModeling

Containers

A container is a FILE format NOT a code scheme

Eg AVI amp WAV are container formats Formats for storage Independent of encoding or content

Others Ogg ASF QuickTime RealMedia Matroska

DivX and MP4

copy DJ Foreman 2012 27

28

Movie (metadata)

Track

Media

Media Information

Container file format (MP4QT)

Chunk map

Comp time

Chunk offset

Key frame

Desc

Decode time

Priority map

Size map

Sample Table

Media Handler

Data information

Media header

Video header

Track header

Movie header

copy DJ Foreman 2012

Pixels A pixel is 3 color-dots (squares) in a collection of dots

(squares) making up the picture Each color-dot is for one of the three primary colors

RGB Minimum of 1 byte per color (depending on bit-depth

which varies with camera) thus at least 3 bytes per pixel

A 10MP camera (30 million color-dots) needs 30M bytes This gets compressed by the camera (JPEG format) Final file size = 30 million bytes divided by the amount

of compression (compression ratio) An 81 compression ratio would reduce that 30 million

bytes to 375 megabytes (as done on a 4 MP camera) A RAW file has NO compression

copy DJ Foreman 2012 29

Image creation

Basic mechanisms Photography

Film Digital ndashPhotoshop Gimp

Drawing Line BrushModeling

Do it yourself OpenGL (2x and 3x)

Modelingrendering programs Breeze designer Ayam 3DCanvas Makehuman Alice

copy DJ Foreman 2012 30

Photography (film) Silver halide crystals (AgCl AgBr AgI)

Exposure to light turns them black Developer removes the loose (exposed) halide leaving

pure silver (a negative image) Fixing bath flushes out unexposed AgHalide

Greatest desire of photographers - a medium with High speed (gathers light quickly) Low noise (no undesired artifacts) High resolution (very detailed images)

Visible lines per mm (lpm) at a specified contrast level

Notes Larger ldquoclumpsrdquo are faster but limit resolution Smaller ldquoclumpsrdquo are hard to coat evenly Randomness of coating prevents Moireacute patternshttpenwikipediaorgwikiMoirC3A9_pattern

copy DJ Foreman 2012 31

FYI ndash a little chemistry

Ag+ X- + hv rarr Ag+ + X + e-

1 Radiation frees the Br2 1 Ag ion + 1 electron (Ag+ + e- rarr Ag0) -gt 1 Ag atom3 Neighbors are now ldquosensitizedrdquo 4 Developing (Reducing) converts exposed AgX to

black metallic Ag and free X5 Remaining AgX and free X are washed away (after

mixing with thiosulfate to make it soluble)

This is one silver (Ag) ldquograinrdquo in the ldquolatent imagerdquo X is any halide (Br Cl etc)

copy DJ Foreman 2012 32

crystal

Radiation (light)

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

7

Vector formats

AI (Adobe Illustrator) CDR (CorelDRAW) CMX (Corel Exchange) CGM (Computer Graphics Metafile ) DXF (AutoCAD ) WMF (Windows Metafile )

copy DJ Foreman 2012

8copy DJ Foreman 2012

Compression

Codecs A device or program for encoding andor

decoding a digital data stream or signal Encoding A-gtD for storage or transmission DecodingD-gtA for playback Lossy vs lossless Raw uncompressed Pulse Code Modulation

(PCM) audio a digital representation of an analog signal where the magnitude of the signal is sampled regularly at uniform intervals Eg (441 kHz 16 bit stereo as represented on an

audio CD or in a wav or aiff file) is a standard across multiple platforms

copy DJ Foreman 2012 9

Codecs - 2

May emphasize aspects of data Video

motion vs color Audio

latency (cell phones) vs high-fidelity (music) Data size

transmission speed or storage size vs data loss

copy DJ Foreman 2012 10

Codec References

Forensic Computing A Practitioners Guide by T Sammes amp B Jenkinson (Springer 2000) ISBN 9781852332990

httpwwwgarykesslernetlibraryfile_sigshtml

copy DJ Foreman 2012 11

Considerations

Tradeoffs compression speed compressed data size quality (data loss)

Areas of study information theory rate-distortion theory

Popular algorithms (all lossless) Lempel-Ziv (LZ) LZW (fast decompression) LZR (LZ-Renau) (ZIP files) LZW (Lempel-Ziv-Welch) (for GIF files)

copy DJ Foreman 2012 12

Lossless Encoding

Huffman compression Bit string representing any symbol is never a

prefix of the bit string representing any other symbolie if 010 is the code for a symbol then no other symbol starts with 010

Frequency table must be known computable or included

Lossless ndash every character gets encoded Arithmetic encoding

Probabilistic algorithm Slightly superior to Huffman but often

patentedcopy DJ Foreman 2012 13

Methodology

Compute a model of the data eg a Huffman tree based on probability Probability determined from input file

Map the data according to the model Read 1 symbol Search the model (tree) for that symbol

If itrsquos a Huffman tree going left=0 right=1 Append each 0 or 1 to the code string When symbol is found you are done Note all symbols will have lt 8 bits (thus

compression)

copy DJ Foreman 2012 14

15

Huffman lab ndash more details Read write are always in bytes

You may need to pad You may have extraneous letters in the output

due to padding You donrsquot need to add a header to specify size

You have 2 weeks to do the lab Code may be in C C++ or Java The readme specifies how to compile the

program Encoded data is sent to a binary file or a simple

text file (but isnrsquot ldquoreadablerdquo as text) Encoded data can only be ldquocheckedrdquo with a

decodercopy DJ Foreman 2012

16

Sample of prefix coding amp compression

copy DJ Foreman 2012

B e a r c a t s (8 bytes of text)010101100110001110001111 encoded as123456781234567812345678 3 bytes

A simple Huffman-like code(wo probability basis) 0 1

A 110

B 010

C 0001

E 10

R 011

S 111

T 001

Compression-Model types

Static Read all the data Compute frequencies Re-read the data amp encode

Dynamic Build simple basic model Modify model as more data is processed

copy DJ Foreman 2012 17

Standard Compressed-File Content

Three parts 24-byte header with magic (defines

codec) variable-length annotation block contiguous segment of audio data

Storage methodology network (big-endian) byte order multi-byte audio data may require byte

reversal in order to operate on it by the arithmetic unit of certain processors

copy DJ Foreman 2012 18

Example 1 Header file for AU files

copy DJ Foreman 2012 19

typedef unsigned long u_32 unsigned 32-bit integer

typedef struct

u_32 magic the ldquomagic numberrdquo

u_32 hdr_size byte offset to start of data

u_32 data_size length (optional)

u_32 encoding data encoding enumeration

u_32 sample_rate samples per second

u_32 channels of interleaved channels

Audio_filehdr

Following is from include ltmultimedialibaudiohgt

AUDIO_FILE_MAGIC ((u_32)0x2e736e64) ldquosndrdquo

Audio Encoding EnumerationsAUDIO_FILE_ENCODING_MULAW_8 (1) 8-bit ISDN u-law

AUDIO_FILE_ENCODING_LINEAR_8 (2) 8-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_16 (3) 16-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_32 (5) 32-bit linear PCM

AUDIO_FILE_ENCODING_FLOAT (6) 32-bit IEEE floating point

AUDIO_FILE_ENCODING_DOUBLE (7) 64-bit IEEE floating point

AUDIO_FILE_ENCODING_ADPCM_G721 (23) 4-bit CCITT g721 ADPCM

AUDIO_FILE_ENCODING_ADPCM_G723_3 (25) CCITT g723 3-bit ADPCM

AUDIO_FILE_ENCODING_ALAW_8 (27) 8-bit ISDN A-law

copy DJ Foreman 2012 20

ldquoLinearrdquo values are SIGNED intrsquosFloats are signed zero-centered normalized to ( -10 lt= x lt= 10 )

Example 2 ZIP files (PkWare)Purpose Size in bytes

Signature header 4

Required version 2

GP flags 2

Method 2

Last mod time 2

Last mod date 2

CRC-32 4

Compressed size 4

Uncompressed size 4

Filename length 2

Extra field length 2

File name variable

extra variable

copy DJ Foreman 2012 21

Ref httplivedocsadobecomflex3htmlhelphtmlcontent=ByteArrays_3html

More info on encoding httpwwwpkwarecomdocumentscase

studiesAPPNOTETXT

httpwwwfileinfocomfiletypescompressed

copy DJ Foreman 2012 22

Recording Bit-Rate

Constant (CBR) rate at which a codecs output data should be

consumed is constant Max bit-rate matters not the average Uses all available bandwidth Not good for storage (lossy)

Variable (VBR) Quantity of datatime unit (for output) varies Better quality for audio and video Slower to encode Supported by most portable devices post 2006

copy DJ Foreman 2012 23

Lab assignment ndash 1 ndash part 1

Write a program with 1 parameter If you are on an encoding team

Open a test file (eg ldquoxyztxtrdquo) Compress it using simple Huffman compression using

the frequency table from my FTP site Output compressed file (eg ldquoxyzencrdquo) to SAME folder

as input

If you are on a decoding team Open compressed file ldquoxyzencrdquo Decompress the input file Output file ldquoxyzdecrdquo so it can be compared to ldquoxyztxtrdquo

The parameter is the full input file path amp name (ie file names MUST NOT be hard-coded)

Remember a code can be a single bit

copy DJ Foreman 2012 24

25

Lab assignment 1 ndash part 2

Use ONLY MY TABLE for the frequencies Use ONLY uppercase letters of the table Do NOT use the special character

tables Find someone in class with the

opposite program and TEST your program to see if it properly encodes or decodes a message

Grading tree construction tree following proper binary output (encoders) proper character output (decoders)

copy DJ Foreman 2012

Working with ImagesModeling

Containers

A container is a FILE format NOT a code scheme

Eg AVI amp WAV are container formats Formats for storage Independent of encoding or content

Others Ogg ASF QuickTime RealMedia Matroska

DivX and MP4

copy DJ Foreman 2012 27

28

Movie (metadata)

Track

Media

Media Information

Container file format (MP4QT)

Chunk map

Comp time

Chunk offset

Key frame

Desc

Decode time

Priority map

Size map

Sample Table

Media Handler

Data information

Media header

Video header

Track header

Movie header

copy DJ Foreman 2012

Pixels A pixel is 3 color-dots (squares) in a collection of dots

(squares) making up the picture Each color-dot is for one of the three primary colors

RGB Minimum of 1 byte per color (depending on bit-depth

which varies with camera) thus at least 3 bytes per pixel

A 10MP camera (30 million color-dots) needs 30M bytes This gets compressed by the camera (JPEG format) Final file size = 30 million bytes divided by the amount

of compression (compression ratio) An 81 compression ratio would reduce that 30 million

bytes to 375 megabytes (as done on a 4 MP camera) A RAW file has NO compression

copy DJ Foreman 2012 29

Image creation

Basic mechanisms Photography

Film Digital ndashPhotoshop Gimp

Drawing Line BrushModeling

Do it yourself OpenGL (2x and 3x)

Modelingrendering programs Breeze designer Ayam 3DCanvas Makehuman Alice

copy DJ Foreman 2012 30

Photography (film) Silver halide crystals (AgCl AgBr AgI)

Exposure to light turns them black Developer removes the loose (exposed) halide leaving

pure silver (a negative image) Fixing bath flushes out unexposed AgHalide

Greatest desire of photographers - a medium with High speed (gathers light quickly) Low noise (no undesired artifacts) High resolution (very detailed images)

Visible lines per mm (lpm) at a specified contrast level

Notes Larger ldquoclumpsrdquo are faster but limit resolution Smaller ldquoclumpsrdquo are hard to coat evenly Randomness of coating prevents Moireacute patternshttpenwikipediaorgwikiMoirC3A9_pattern

copy DJ Foreman 2012 31

FYI ndash a little chemistry

Ag+ X- + hv rarr Ag+ + X + e-

1 Radiation frees the Br2 1 Ag ion + 1 electron (Ag+ + e- rarr Ag0) -gt 1 Ag atom3 Neighbors are now ldquosensitizedrdquo 4 Developing (Reducing) converts exposed AgX to

black metallic Ag and free X5 Remaining AgX and free X are washed away (after

mixing with thiosulfate to make it soluble)

This is one silver (Ag) ldquograinrdquo in the ldquolatent imagerdquo X is any halide (Br Cl etc)

copy DJ Foreman 2012 32

crystal

Radiation (light)

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

8copy DJ Foreman 2012

Compression

Codecs A device or program for encoding andor

decoding a digital data stream or signal Encoding A-gtD for storage or transmission DecodingD-gtA for playback Lossy vs lossless Raw uncompressed Pulse Code Modulation

(PCM) audio a digital representation of an analog signal where the magnitude of the signal is sampled regularly at uniform intervals Eg (441 kHz 16 bit stereo as represented on an

audio CD or in a wav or aiff file) is a standard across multiple platforms

copy DJ Foreman 2012 9

Codecs - 2

May emphasize aspects of data Video

motion vs color Audio

latency (cell phones) vs high-fidelity (music) Data size

transmission speed or storage size vs data loss

copy DJ Foreman 2012 10

Codec References

Forensic Computing A Practitioners Guide by T Sammes amp B Jenkinson (Springer 2000) ISBN 9781852332990

httpwwwgarykesslernetlibraryfile_sigshtml

copy DJ Foreman 2012 11

Considerations

Tradeoffs compression speed compressed data size quality (data loss)

Areas of study information theory rate-distortion theory

Popular algorithms (all lossless) Lempel-Ziv (LZ) LZW (fast decompression) LZR (LZ-Renau) (ZIP files) LZW (Lempel-Ziv-Welch) (for GIF files)

copy DJ Foreman 2012 12

Lossless Encoding

Huffman compression Bit string representing any symbol is never a

prefix of the bit string representing any other symbolie if 010 is the code for a symbol then no other symbol starts with 010

Frequency table must be known computable or included

Lossless ndash every character gets encoded Arithmetic encoding

Probabilistic algorithm Slightly superior to Huffman but often

patentedcopy DJ Foreman 2012 13

Methodology

Compute a model of the data eg a Huffman tree based on probability Probability determined from input file

Map the data according to the model Read 1 symbol Search the model (tree) for that symbol

If itrsquos a Huffman tree going left=0 right=1 Append each 0 or 1 to the code string When symbol is found you are done Note all symbols will have lt 8 bits (thus

compression)

copy DJ Foreman 2012 14

15

Huffman lab ndash more details Read write are always in bytes

You may need to pad You may have extraneous letters in the output

due to padding You donrsquot need to add a header to specify size

You have 2 weeks to do the lab Code may be in C C++ or Java The readme specifies how to compile the

program Encoded data is sent to a binary file or a simple

text file (but isnrsquot ldquoreadablerdquo as text) Encoded data can only be ldquocheckedrdquo with a

decodercopy DJ Foreman 2012

16

Sample of prefix coding amp compression

copy DJ Foreman 2012

B e a r c a t s (8 bytes of text)010101100110001110001111 encoded as123456781234567812345678 3 bytes

A simple Huffman-like code(wo probability basis) 0 1

A 110

B 010

C 0001

E 10

R 011

S 111

T 001

Compression-Model types

Static Read all the data Compute frequencies Re-read the data amp encode

Dynamic Build simple basic model Modify model as more data is processed

copy DJ Foreman 2012 17

Standard Compressed-File Content

Three parts 24-byte header with magic (defines

codec) variable-length annotation block contiguous segment of audio data

Storage methodology network (big-endian) byte order multi-byte audio data may require byte

reversal in order to operate on it by the arithmetic unit of certain processors

copy DJ Foreman 2012 18

Example 1 Header file for AU files

copy DJ Foreman 2012 19

typedef unsigned long u_32 unsigned 32-bit integer

typedef struct

u_32 magic the ldquomagic numberrdquo

u_32 hdr_size byte offset to start of data

u_32 data_size length (optional)

u_32 encoding data encoding enumeration

u_32 sample_rate samples per second

u_32 channels of interleaved channels

Audio_filehdr

Following is from include ltmultimedialibaudiohgt

AUDIO_FILE_MAGIC ((u_32)0x2e736e64) ldquosndrdquo

Audio Encoding EnumerationsAUDIO_FILE_ENCODING_MULAW_8 (1) 8-bit ISDN u-law

AUDIO_FILE_ENCODING_LINEAR_8 (2) 8-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_16 (3) 16-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_32 (5) 32-bit linear PCM

AUDIO_FILE_ENCODING_FLOAT (6) 32-bit IEEE floating point

AUDIO_FILE_ENCODING_DOUBLE (7) 64-bit IEEE floating point

AUDIO_FILE_ENCODING_ADPCM_G721 (23) 4-bit CCITT g721 ADPCM

AUDIO_FILE_ENCODING_ADPCM_G723_3 (25) CCITT g723 3-bit ADPCM

AUDIO_FILE_ENCODING_ALAW_8 (27) 8-bit ISDN A-law

copy DJ Foreman 2012 20

ldquoLinearrdquo values are SIGNED intrsquosFloats are signed zero-centered normalized to ( -10 lt= x lt= 10 )

Example 2 ZIP files (PkWare)Purpose Size in bytes

Signature header 4

Required version 2

GP flags 2

Method 2

Last mod time 2

Last mod date 2

CRC-32 4

Compressed size 4

Uncompressed size 4

Filename length 2

Extra field length 2

File name variable

extra variable

copy DJ Foreman 2012 21

Ref httplivedocsadobecomflex3htmlhelphtmlcontent=ByteArrays_3html

More info on encoding httpwwwpkwarecomdocumentscase

studiesAPPNOTETXT

httpwwwfileinfocomfiletypescompressed

copy DJ Foreman 2012 22

Recording Bit-Rate

Constant (CBR) rate at which a codecs output data should be

consumed is constant Max bit-rate matters not the average Uses all available bandwidth Not good for storage (lossy)

Variable (VBR) Quantity of datatime unit (for output) varies Better quality for audio and video Slower to encode Supported by most portable devices post 2006

copy DJ Foreman 2012 23

Lab assignment ndash 1 ndash part 1

Write a program with 1 parameter If you are on an encoding team

Open a test file (eg ldquoxyztxtrdquo) Compress it using simple Huffman compression using

the frequency table from my FTP site Output compressed file (eg ldquoxyzencrdquo) to SAME folder

as input

If you are on a decoding team Open compressed file ldquoxyzencrdquo Decompress the input file Output file ldquoxyzdecrdquo so it can be compared to ldquoxyztxtrdquo

The parameter is the full input file path amp name (ie file names MUST NOT be hard-coded)

Remember a code can be a single bit

copy DJ Foreman 2012 24

25

Lab assignment 1 ndash part 2

Use ONLY MY TABLE for the frequencies Use ONLY uppercase letters of the table Do NOT use the special character

tables Find someone in class with the

opposite program and TEST your program to see if it properly encodes or decodes a message

Grading tree construction tree following proper binary output (encoders) proper character output (decoders)

copy DJ Foreman 2012

Working with ImagesModeling

Containers

A container is a FILE format NOT a code scheme

Eg AVI amp WAV are container formats Formats for storage Independent of encoding or content

Others Ogg ASF QuickTime RealMedia Matroska

DivX and MP4

copy DJ Foreman 2012 27

28

Movie (metadata)

Track

Media

Media Information

Container file format (MP4QT)

Chunk map

Comp time

Chunk offset

Key frame

Desc

Decode time

Priority map

Size map

Sample Table

Media Handler

Data information

Media header

Video header

Track header

Movie header

copy DJ Foreman 2012

Pixels A pixel is 3 color-dots (squares) in a collection of dots

(squares) making up the picture Each color-dot is for one of the three primary colors

RGB Minimum of 1 byte per color (depending on bit-depth

which varies with camera) thus at least 3 bytes per pixel

A 10MP camera (30 million color-dots) needs 30M bytes This gets compressed by the camera (JPEG format) Final file size = 30 million bytes divided by the amount

of compression (compression ratio) An 81 compression ratio would reduce that 30 million

bytes to 375 megabytes (as done on a 4 MP camera) A RAW file has NO compression

copy DJ Foreman 2012 29

Image creation

Basic mechanisms Photography

Film Digital ndashPhotoshop Gimp

Drawing Line BrushModeling

Do it yourself OpenGL (2x and 3x)

Modelingrendering programs Breeze designer Ayam 3DCanvas Makehuman Alice

copy DJ Foreman 2012 30

Photography (film) Silver halide crystals (AgCl AgBr AgI)

Exposure to light turns them black Developer removes the loose (exposed) halide leaving

pure silver (a negative image) Fixing bath flushes out unexposed AgHalide

Greatest desire of photographers - a medium with High speed (gathers light quickly) Low noise (no undesired artifacts) High resolution (very detailed images)

Visible lines per mm (lpm) at a specified contrast level

Notes Larger ldquoclumpsrdquo are faster but limit resolution Smaller ldquoclumpsrdquo are hard to coat evenly Randomness of coating prevents Moireacute patternshttpenwikipediaorgwikiMoirC3A9_pattern

copy DJ Foreman 2012 31

FYI ndash a little chemistry

Ag+ X- + hv rarr Ag+ + X + e-

1 Radiation frees the Br2 1 Ag ion + 1 electron (Ag+ + e- rarr Ag0) -gt 1 Ag atom3 Neighbors are now ldquosensitizedrdquo 4 Developing (Reducing) converts exposed AgX to

black metallic Ag and free X5 Remaining AgX and free X are washed away (after

mixing with thiosulfate to make it soluble)

This is one silver (Ag) ldquograinrdquo in the ldquolatent imagerdquo X is any halide (Br Cl etc)

copy DJ Foreman 2012 32

crystal

Radiation (light)

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Codecs A device or program for encoding andor

decoding a digital data stream or signal Encoding A-gtD for storage or transmission DecodingD-gtA for playback Lossy vs lossless Raw uncompressed Pulse Code Modulation

(PCM) audio a digital representation of an analog signal where the magnitude of the signal is sampled regularly at uniform intervals Eg (441 kHz 16 bit stereo as represented on an

audio CD or in a wav or aiff file) is a standard across multiple platforms

copy DJ Foreman 2012 9

Codecs - 2

May emphasize aspects of data Video

motion vs color Audio

latency (cell phones) vs high-fidelity (music) Data size

transmission speed or storage size vs data loss

copy DJ Foreman 2012 10

Codec References

Forensic Computing A Practitioners Guide by T Sammes amp B Jenkinson (Springer 2000) ISBN 9781852332990

httpwwwgarykesslernetlibraryfile_sigshtml

copy DJ Foreman 2012 11

Considerations

Tradeoffs compression speed compressed data size quality (data loss)

Areas of study information theory rate-distortion theory

Popular algorithms (all lossless) Lempel-Ziv (LZ) LZW (fast decompression) LZR (LZ-Renau) (ZIP files) LZW (Lempel-Ziv-Welch) (for GIF files)

copy DJ Foreman 2012 12

Lossless Encoding

Huffman compression Bit string representing any symbol is never a

prefix of the bit string representing any other symbolie if 010 is the code for a symbol then no other symbol starts with 010

Frequency table must be known computable or included

Lossless ndash every character gets encoded Arithmetic encoding

Probabilistic algorithm Slightly superior to Huffman but often

patentedcopy DJ Foreman 2012 13

Methodology

Compute a model of the data eg a Huffman tree based on probability Probability determined from input file

Map the data according to the model Read 1 symbol Search the model (tree) for that symbol

If itrsquos a Huffman tree going left=0 right=1 Append each 0 or 1 to the code string When symbol is found you are done Note all symbols will have lt 8 bits (thus

compression)

copy DJ Foreman 2012 14

15

Huffman lab ndash more details Read write are always in bytes

You may need to pad You may have extraneous letters in the output

due to padding You donrsquot need to add a header to specify size

You have 2 weeks to do the lab Code may be in C C++ or Java The readme specifies how to compile the

program Encoded data is sent to a binary file or a simple

text file (but isnrsquot ldquoreadablerdquo as text) Encoded data can only be ldquocheckedrdquo with a

decodercopy DJ Foreman 2012

16

Sample of prefix coding amp compression

copy DJ Foreman 2012

B e a r c a t s (8 bytes of text)010101100110001110001111 encoded as123456781234567812345678 3 bytes

A simple Huffman-like code(wo probability basis) 0 1

A 110

B 010

C 0001

E 10

R 011

S 111

T 001

Compression-Model types

Static Read all the data Compute frequencies Re-read the data amp encode

Dynamic Build simple basic model Modify model as more data is processed

copy DJ Foreman 2012 17

Standard Compressed-File Content

Three parts 24-byte header with magic (defines

codec) variable-length annotation block contiguous segment of audio data

Storage methodology network (big-endian) byte order multi-byte audio data may require byte

reversal in order to operate on it by the arithmetic unit of certain processors

copy DJ Foreman 2012 18

Example 1 Header file for AU files

copy DJ Foreman 2012 19

typedef unsigned long u_32 unsigned 32-bit integer

typedef struct

u_32 magic the ldquomagic numberrdquo

u_32 hdr_size byte offset to start of data

u_32 data_size length (optional)

u_32 encoding data encoding enumeration

u_32 sample_rate samples per second

u_32 channels of interleaved channels

Audio_filehdr

Following is from include ltmultimedialibaudiohgt

AUDIO_FILE_MAGIC ((u_32)0x2e736e64) ldquosndrdquo

Audio Encoding EnumerationsAUDIO_FILE_ENCODING_MULAW_8 (1) 8-bit ISDN u-law

AUDIO_FILE_ENCODING_LINEAR_8 (2) 8-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_16 (3) 16-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_32 (5) 32-bit linear PCM

AUDIO_FILE_ENCODING_FLOAT (6) 32-bit IEEE floating point

AUDIO_FILE_ENCODING_DOUBLE (7) 64-bit IEEE floating point

AUDIO_FILE_ENCODING_ADPCM_G721 (23) 4-bit CCITT g721 ADPCM

AUDIO_FILE_ENCODING_ADPCM_G723_3 (25) CCITT g723 3-bit ADPCM

AUDIO_FILE_ENCODING_ALAW_8 (27) 8-bit ISDN A-law

copy DJ Foreman 2012 20

ldquoLinearrdquo values are SIGNED intrsquosFloats are signed zero-centered normalized to ( -10 lt= x lt= 10 )

Example 2 ZIP files (PkWare)Purpose Size in bytes

Signature header 4

Required version 2

GP flags 2

Method 2

Last mod time 2

Last mod date 2

CRC-32 4

Compressed size 4

Uncompressed size 4

Filename length 2

Extra field length 2

File name variable

extra variable

copy DJ Foreman 2012 21

Ref httplivedocsadobecomflex3htmlhelphtmlcontent=ByteArrays_3html

More info on encoding httpwwwpkwarecomdocumentscase

studiesAPPNOTETXT

httpwwwfileinfocomfiletypescompressed

copy DJ Foreman 2012 22

Recording Bit-Rate

Constant (CBR) rate at which a codecs output data should be

consumed is constant Max bit-rate matters not the average Uses all available bandwidth Not good for storage (lossy)

Variable (VBR) Quantity of datatime unit (for output) varies Better quality for audio and video Slower to encode Supported by most portable devices post 2006

copy DJ Foreman 2012 23

Lab assignment ndash 1 ndash part 1

Write a program with 1 parameter If you are on an encoding team

Open a test file (eg ldquoxyztxtrdquo) Compress it using simple Huffman compression using

the frequency table from my FTP site Output compressed file (eg ldquoxyzencrdquo) to SAME folder

as input

If you are on a decoding team Open compressed file ldquoxyzencrdquo Decompress the input file Output file ldquoxyzdecrdquo so it can be compared to ldquoxyztxtrdquo

The parameter is the full input file path amp name (ie file names MUST NOT be hard-coded)

Remember a code can be a single bit

copy DJ Foreman 2012 24

25

Lab assignment 1 ndash part 2

Use ONLY MY TABLE for the frequencies Use ONLY uppercase letters of the table Do NOT use the special character

tables Find someone in class with the

opposite program and TEST your program to see if it properly encodes or decodes a message

Grading tree construction tree following proper binary output (encoders) proper character output (decoders)

copy DJ Foreman 2012

Working with ImagesModeling

Containers

A container is a FILE format NOT a code scheme

Eg AVI amp WAV are container formats Formats for storage Independent of encoding or content

Others Ogg ASF QuickTime RealMedia Matroska

DivX and MP4

copy DJ Foreman 2012 27

28

Movie (metadata)

Track

Media

Media Information

Container file format (MP4QT)

Chunk map

Comp time

Chunk offset

Key frame

Desc

Decode time

Priority map

Size map

Sample Table

Media Handler

Data information

Media header

Video header

Track header

Movie header

copy DJ Foreman 2012

Pixels A pixel is 3 color-dots (squares) in a collection of dots

(squares) making up the picture Each color-dot is for one of the three primary colors

RGB Minimum of 1 byte per color (depending on bit-depth

which varies with camera) thus at least 3 bytes per pixel

A 10MP camera (30 million color-dots) needs 30M bytes This gets compressed by the camera (JPEG format) Final file size = 30 million bytes divided by the amount

of compression (compression ratio) An 81 compression ratio would reduce that 30 million

bytes to 375 megabytes (as done on a 4 MP camera) A RAW file has NO compression

copy DJ Foreman 2012 29

Image creation

Basic mechanisms Photography

Film Digital ndashPhotoshop Gimp

Drawing Line BrushModeling

Do it yourself OpenGL (2x and 3x)

Modelingrendering programs Breeze designer Ayam 3DCanvas Makehuman Alice

copy DJ Foreman 2012 30

Photography (film) Silver halide crystals (AgCl AgBr AgI)

Exposure to light turns them black Developer removes the loose (exposed) halide leaving

pure silver (a negative image) Fixing bath flushes out unexposed AgHalide

Greatest desire of photographers - a medium with High speed (gathers light quickly) Low noise (no undesired artifacts) High resolution (very detailed images)

Visible lines per mm (lpm) at a specified contrast level

Notes Larger ldquoclumpsrdquo are faster but limit resolution Smaller ldquoclumpsrdquo are hard to coat evenly Randomness of coating prevents Moireacute patternshttpenwikipediaorgwikiMoirC3A9_pattern

copy DJ Foreman 2012 31

FYI ndash a little chemistry

Ag+ X- + hv rarr Ag+ + X + e-

1 Radiation frees the Br2 1 Ag ion + 1 electron (Ag+ + e- rarr Ag0) -gt 1 Ag atom3 Neighbors are now ldquosensitizedrdquo 4 Developing (Reducing) converts exposed AgX to

black metallic Ag and free X5 Remaining AgX and free X are washed away (after

mixing with thiosulfate to make it soluble)

This is one silver (Ag) ldquograinrdquo in the ldquolatent imagerdquo X is any halide (Br Cl etc)

copy DJ Foreman 2012 32

crystal

Radiation (light)

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Codecs - 2

May emphasize aspects of data Video

motion vs color Audio

latency (cell phones) vs high-fidelity (music) Data size

transmission speed or storage size vs data loss

copy DJ Foreman 2012 10

Codec References

Forensic Computing A Practitioners Guide by T Sammes amp B Jenkinson (Springer 2000) ISBN 9781852332990

httpwwwgarykesslernetlibraryfile_sigshtml

copy DJ Foreman 2012 11

Considerations

Tradeoffs compression speed compressed data size quality (data loss)

Areas of study information theory rate-distortion theory

Popular algorithms (all lossless) Lempel-Ziv (LZ) LZW (fast decompression) LZR (LZ-Renau) (ZIP files) LZW (Lempel-Ziv-Welch) (for GIF files)

copy DJ Foreman 2012 12

Lossless Encoding

Huffman compression Bit string representing any symbol is never a

prefix of the bit string representing any other symbolie if 010 is the code for a symbol then no other symbol starts with 010

Frequency table must be known computable or included

Lossless ndash every character gets encoded Arithmetic encoding

Probabilistic algorithm Slightly superior to Huffman but often

patentedcopy DJ Foreman 2012 13

Methodology

Compute a model of the data eg a Huffman tree based on probability Probability determined from input file

Map the data according to the model Read 1 symbol Search the model (tree) for that symbol

If itrsquos a Huffman tree going left=0 right=1 Append each 0 or 1 to the code string When symbol is found you are done Note all symbols will have lt 8 bits (thus

compression)

copy DJ Foreman 2012 14

15

Huffman lab ndash more details Read write are always in bytes

You may need to pad You may have extraneous letters in the output

due to padding You donrsquot need to add a header to specify size

You have 2 weeks to do the lab Code may be in C C++ or Java The readme specifies how to compile the

program Encoded data is sent to a binary file or a simple

text file (but isnrsquot ldquoreadablerdquo as text) Encoded data can only be ldquocheckedrdquo with a

decodercopy DJ Foreman 2012

16

Sample of prefix coding amp compression

copy DJ Foreman 2012

B e a r c a t s (8 bytes of text)010101100110001110001111 encoded as123456781234567812345678 3 bytes

A simple Huffman-like code(wo probability basis) 0 1

A 110

B 010

C 0001

E 10

R 011

S 111

T 001

Compression-Model types

Static Read all the data Compute frequencies Re-read the data amp encode

Dynamic Build simple basic model Modify model as more data is processed

copy DJ Foreman 2012 17

Standard Compressed-File Content

Three parts 24-byte header with magic (defines

codec) variable-length annotation block contiguous segment of audio data

Storage methodology network (big-endian) byte order multi-byte audio data may require byte

reversal in order to operate on it by the arithmetic unit of certain processors

copy DJ Foreman 2012 18

Example 1 Header file for AU files

copy DJ Foreman 2012 19

typedef unsigned long u_32 unsigned 32-bit integer

typedef struct

u_32 magic the ldquomagic numberrdquo

u_32 hdr_size byte offset to start of data

u_32 data_size length (optional)

u_32 encoding data encoding enumeration

u_32 sample_rate samples per second

u_32 channels of interleaved channels

Audio_filehdr

Following is from include ltmultimedialibaudiohgt

AUDIO_FILE_MAGIC ((u_32)0x2e736e64) ldquosndrdquo

Audio Encoding EnumerationsAUDIO_FILE_ENCODING_MULAW_8 (1) 8-bit ISDN u-law

AUDIO_FILE_ENCODING_LINEAR_8 (2) 8-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_16 (3) 16-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_32 (5) 32-bit linear PCM

AUDIO_FILE_ENCODING_FLOAT (6) 32-bit IEEE floating point

AUDIO_FILE_ENCODING_DOUBLE (7) 64-bit IEEE floating point

AUDIO_FILE_ENCODING_ADPCM_G721 (23) 4-bit CCITT g721 ADPCM

AUDIO_FILE_ENCODING_ADPCM_G723_3 (25) CCITT g723 3-bit ADPCM

AUDIO_FILE_ENCODING_ALAW_8 (27) 8-bit ISDN A-law

copy DJ Foreman 2012 20

ldquoLinearrdquo values are SIGNED intrsquosFloats are signed zero-centered normalized to ( -10 lt= x lt= 10 )

Example 2 ZIP files (PkWare)Purpose Size in bytes

Signature header 4

Required version 2

GP flags 2

Method 2

Last mod time 2

Last mod date 2

CRC-32 4

Compressed size 4

Uncompressed size 4

Filename length 2

Extra field length 2

File name variable

extra variable

copy DJ Foreman 2012 21

Ref httplivedocsadobecomflex3htmlhelphtmlcontent=ByteArrays_3html

More info on encoding httpwwwpkwarecomdocumentscase

studiesAPPNOTETXT

httpwwwfileinfocomfiletypescompressed

copy DJ Foreman 2012 22

Recording Bit-Rate

Constant (CBR) rate at which a codecs output data should be

consumed is constant Max bit-rate matters not the average Uses all available bandwidth Not good for storage (lossy)

Variable (VBR) Quantity of datatime unit (for output) varies Better quality for audio and video Slower to encode Supported by most portable devices post 2006

copy DJ Foreman 2012 23

Lab assignment ndash 1 ndash part 1

Write a program with 1 parameter If you are on an encoding team

Open a test file (eg ldquoxyztxtrdquo) Compress it using simple Huffman compression using

the frequency table from my FTP site Output compressed file (eg ldquoxyzencrdquo) to SAME folder

as input

If you are on a decoding team Open compressed file ldquoxyzencrdquo Decompress the input file Output file ldquoxyzdecrdquo so it can be compared to ldquoxyztxtrdquo

The parameter is the full input file path amp name (ie file names MUST NOT be hard-coded)

Remember a code can be a single bit

copy DJ Foreman 2012 24

25

Lab assignment 1 ndash part 2

Use ONLY MY TABLE for the frequencies Use ONLY uppercase letters of the table Do NOT use the special character

tables Find someone in class with the

opposite program and TEST your program to see if it properly encodes or decodes a message

Grading tree construction tree following proper binary output (encoders) proper character output (decoders)

copy DJ Foreman 2012

Working with ImagesModeling

Containers

A container is a FILE format NOT a code scheme

Eg AVI amp WAV are container formats Formats for storage Independent of encoding or content

Others Ogg ASF QuickTime RealMedia Matroska

DivX and MP4

copy DJ Foreman 2012 27

28

Movie (metadata)

Track

Media

Media Information

Container file format (MP4QT)

Chunk map

Comp time

Chunk offset

Key frame

Desc

Decode time

Priority map

Size map

Sample Table

Media Handler

Data information

Media header

Video header

Track header

Movie header

copy DJ Foreman 2012

Pixels A pixel is 3 color-dots (squares) in a collection of dots

(squares) making up the picture Each color-dot is for one of the three primary colors

RGB Minimum of 1 byte per color (depending on bit-depth

which varies with camera) thus at least 3 bytes per pixel

A 10MP camera (30 million color-dots) needs 30M bytes This gets compressed by the camera (JPEG format) Final file size = 30 million bytes divided by the amount

of compression (compression ratio) An 81 compression ratio would reduce that 30 million

bytes to 375 megabytes (as done on a 4 MP camera) A RAW file has NO compression

copy DJ Foreman 2012 29

Image creation

Basic mechanisms Photography

Film Digital ndashPhotoshop Gimp

Drawing Line BrushModeling

Do it yourself OpenGL (2x and 3x)

Modelingrendering programs Breeze designer Ayam 3DCanvas Makehuman Alice

copy DJ Foreman 2012 30

Photography (film) Silver halide crystals (AgCl AgBr AgI)

Exposure to light turns them black Developer removes the loose (exposed) halide leaving

pure silver (a negative image) Fixing bath flushes out unexposed AgHalide

Greatest desire of photographers - a medium with High speed (gathers light quickly) Low noise (no undesired artifacts) High resolution (very detailed images)

Visible lines per mm (lpm) at a specified contrast level

Notes Larger ldquoclumpsrdquo are faster but limit resolution Smaller ldquoclumpsrdquo are hard to coat evenly Randomness of coating prevents Moireacute patternshttpenwikipediaorgwikiMoirC3A9_pattern

copy DJ Foreman 2012 31

FYI ndash a little chemistry

Ag+ X- + hv rarr Ag+ + X + e-

1 Radiation frees the Br2 1 Ag ion + 1 electron (Ag+ + e- rarr Ag0) -gt 1 Ag atom3 Neighbors are now ldquosensitizedrdquo 4 Developing (Reducing) converts exposed AgX to

black metallic Ag and free X5 Remaining AgX and free X are washed away (after

mixing with thiosulfate to make it soluble)

This is one silver (Ag) ldquograinrdquo in the ldquolatent imagerdquo X is any halide (Br Cl etc)

copy DJ Foreman 2012 32

crystal

Radiation (light)

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Codec References

Forensic Computing A Practitioners Guide by T Sammes amp B Jenkinson (Springer 2000) ISBN 9781852332990

httpwwwgarykesslernetlibraryfile_sigshtml

copy DJ Foreman 2012 11

Considerations

Tradeoffs compression speed compressed data size quality (data loss)

Areas of study information theory rate-distortion theory

Popular algorithms (all lossless) Lempel-Ziv (LZ) LZW (fast decompression) LZR (LZ-Renau) (ZIP files) LZW (Lempel-Ziv-Welch) (for GIF files)

copy DJ Foreman 2012 12

Lossless Encoding

Huffman compression Bit string representing any symbol is never a

prefix of the bit string representing any other symbolie if 010 is the code for a symbol then no other symbol starts with 010

Frequency table must be known computable or included

Lossless ndash every character gets encoded Arithmetic encoding

Probabilistic algorithm Slightly superior to Huffman but often

patentedcopy DJ Foreman 2012 13

Methodology

Compute a model of the data eg a Huffman tree based on probability Probability determined from input file

Map the data according to the model Read 1 symbol Search the model (tree) for that symbol

If itrsquos a Huffman tree going left=0 right=1 Append each 0 or 1 to the code string When symbol is found you are done Note all symbols will have lt 8 bits (thus

compression)

copy DJ Foreman 2012 14

15

Huffman lab ndash more details Read write are always in bytes

You may need to pad You may have extraneous letters in the output

due to padding You donrsquot need to add a header to specify size

You have 2 weeks to do the lab Code may be in C C++ or Java The readme specifies how to compile the

program Encoded data is sent to a binary file or a simple

text file (but isnrsquot ldquoreadablerdquo as text) Encoded data can only be ldquocheckedrdquo with a

decodercopy DJ Foreman 2012

16

Sample of prefix coding amp compression

copy DJ Foreman 2012

B e a r c a t s (8 bytes of text)010101100110001110001111 encoded as123456781234567812345678 3 bytes

A simple Huffman-like code(wo probability basis) 0 1

A 110

B 010

C 0001

E 10

R 011

S 111

T 001

Compression-Model types

Static Read all the data Compute frequencies Re-read the data amp encode

Dynamic Build simple basic model Modify model as more data is processed

copy DJ Foreman 2012 17

Standard Compressed-File Content

Three parts 24-byte header with magic (defines

codec) variable-length annotation block contiguous segment of audio data

Storage methodology network (big-endian) byte order multi-byte audio data may require byte

reversal in order to operate on it by the arithmetic unit of certain processors

copy DJ Foreman 2012 18

Example 1 Header file for AU files

copy DJ Foreman 2012 19

typedef unsigned long u_32 unsigned 32-bit integer

typedef struct

u_32 magic the ldquomagic numberrdquo

u_32 hdr_size byte offset to start of data

u_32 data_size length (optional)

u_32 encoding data encoding enumeration

u_32 sample_rate samples per second

u_32 channels of interleaved channels

Audio_filehdr

Following is from include ltmultimedialibaudiohgt

AUDIO_FILE_MAGIC ((u_32)0x2e736e64) ldquosndrdquo

Audio Encoding EnumerationsAUDIO_FILE_ENCODING_MULAW_8 (1) 8-bit ISDN u-law

AUDIO_FILE_ENCODING_LINEAR_8 (2) 8-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_16 (3) 16-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_32 (5) 32-bit linear PCM

AUDIO_FILE_ENCODING_FLOAT (6) 32-bit IEEE floating point

AUDIO_FILE_ENCODING_DOUBLE (7) 64-bit IEEE floating point

AUDIO_FILE_ENCODING_ADPCM_G721 (23) 4-bit CCITT g721 ADPCM

AUDIO_FILE_ENCODING_ADPCM_G723_3 (25) CCITT g723 3-bit ADPCM

AUDIO_FILE_ENCODING_ALAW_8 (27) 8-bit ISDN A-law

copy DJ Foreman 2012 20

ldquoLinearrdquo values are SIGNED intrsquosFloats are signed zero-centered normalized to ( -10 lt= x lt= 10 )

Example 2 ZIP files (PkWare)Purpose Size in bytes

Signature header 4

Required version 2

GP flags 2

Method 2

Last mod time 2

Last mod date 2

CRC-32 4

Compressed size 4

Uncompressed size 4

Filename length 2

Extra field length 2

File name variable

extra variable

copy DJ Foreman 2012 21

Ref httplivedocsadobecomflex3htmlhelphtmlcontent=ByteArrays_3html

More info on encoding httpwwwpkwarecomdocumentscase

studiesAPPNOTETXT

httpwwwfileinfocomfiletypescompressed

copy DJ Foreman 2012 22

Recording Bit-Rate

Constant (CBR) rate at which a codecs output data should be

consumed is constant Max bit-rate matters not the average Uses all available bandwidth Not good for storage (lossy)

Variable (VBR) Quantity of datatime unit (for output) varies Better quality for audio and video Slower to encode Supported by most portable devices post 2006

copy DJ Foreman 2012 23

Lab assignment ndash 1 ndash part 1

Write a program with 1 parameter If you are on an encoding team

Open a test file (eg ldquoxyztxtrdquo) Compress it using simple Huffman compression using

the frequency table from my FTP site Output compressed file (eg ldquoxyzencrdquo) to SAME folder

as input

If you are on a decoding team Open compressed file ldquoxyzencrdquo Decompress the input file Output file ldquoxyzdecrdquo so it can be compared to ldquoxyztxtrdquo

The parameter is the full input file path amp name (ie file names MUST NOT be hard-coded)

Remember a code can be a single bit

copy DJ Foreman 2012 24

25

Lab assignment 1 ndash part 2

Use ONLY MY TABLE for the frequencies Use ONLY uppercase letters of the table Do NOT use the special character

tables Find someone in class with the

opposite program and TEST your program to see if it properly encodes or decodes a message

Grading tree construction tree following proper binary output (encoders) proper character output (decoders)

copy DJ Foreman 2012

Working with ImagesModeling

Containers

A container is a FILE format NOT a code scheme

Eg AVI amp WAV are container formats Formats for storage Independent of encoding or content

Others Ogg ASF QuickTime RealMedia Matroska

DivX and MP4

copy DJ Foreman 2012 27

28

Movie (metadata)

Track

Media

Media Information

Container file format (MP4QT)

Chunk map

Comp time

Chunk offset

Key frame

Desc

Decode time

Priority map

Size map

Sample Table

Media Handler

Data information

Media header

Video header

Track header

Movie header

copy DJ Foreman 2012

Pixels A pixel is 3 color-dots (squares) in a collection of dots

(squares) making up the picture Each color-dot is for one of the three primary colors

RGB Minimum of 1 byte per color (depending on bit-depth

which varies with camera) thus at least 3 bytes per pixel

A 10MP camera (30 million color-dots) needs 30M bytes This gets compressed by the camera (JPEG format) Final file size = 30 million bytes divided by the amount

of compression (compression ratio) An 81 compression ratio would reduce that 30 million

bytes to 375 megabytes (as done on a 4 MP camera) A RAW file has NO compression

copy DJ Foreman 2012 29

Image creation

Basic mechanisms Photography

Film Digital ndashPhotoshop Gimp

Drawing Line BrushModeling

Do it yourself OpenGL (2x and 3x)

Modelingrendering programs Breeze designer Ayam 3DCanvas Makehuman Alice

copy DJ Foreman 2012 30

Photography (film) Silver halide crystals (AgCl AgBr AgI)

Exposure to light turns them black Developer removes the loose (exposed) halide leaving

pure silver (a negative image) Fixing bath flushes out unexposed AgHalide

Greatest desire of photographers - a medium with High speed (gathers light quickly) Low noise (no undesired artifacts) High resolution (very detailed images)

Visible lines per mm (lpm) at a specified contrast level

Notes Larger ldquoclumpsrdquo are faster but limit resolution Smaller ldquoclumpsrdquo are hard to coat evenly Randomness of coating prevents Moireacute patternshttpenwikipediaorgwikiMoirC3A9_pattern

copy DJ Foreman 2012 31

FYI ndash a little chemistry

Ag+ X- + hv rarr Ag+ + X + e-

1 Radiation frees the Br2 1 Ag ion + 1 electron (Ag+ + e- rarr Ag0) -gt 1 Ag atom3 Neighbors are now ldquosensitizedrdquo 4 Developing (Reducing) converts exposed AgX to

black metallic Ag and free X5 Remaining AgX and free X are washed away (after

mixing with thiosulfate to make it soluble)

This is one silver (Ag) ldquograinrdquo in the ldquolatent imagerdquo X is any halide (Br Cl etc)

copy DJ Foreman 2012 32

crystal

Radiation (light)

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Considerations

Tradeoffs compression speed compressed data size quality (data loss)

Areas of study information theory rate-distortion theory

Popular algorithms (all lossless) Lempel-Ziv (LZ) LZW (fast decompression) LZR (LZ-Renau) (ZIP files) LZW (Lempel-Ziv-Welch) (for GIF files)

copy DJ Foreman 2012 12

Lossless Encoding

Huffman compression Bit string representing any symbol is never a

prefix of the bit string representing any other symbolie if 010 is the code for a symbol then no other symbol starts with 010

Frequency table must be known computable or included

Lossless ndash every character gets encoded Arithmetic encoding

Probabilistic algorithm Slightly superior to Huffman but often

patentedcopy DJ Foreman 2012 13

Methodology

Compute a model of the data eg a Huffman tree based on probability Probability determined from input file

Map the data according to the model Read 1 symbol Search the model (tree) for that symbol

If itrsquos a Huffman tree going left=0 right=1 Append each 0 or 1 to the code string When symbol is found you are done Note all symbols will have lt 8 bits (thus

compression)

copy DJ Foreman 2012 14

15

Huffman lab ndash more details Read write are always in bytes

You may need to pad You may have extraneous letters in the output

due to padding You donrsquot need to add a header to specify size

You have 2 weeks to do the lab Code may be in C C++ or Java The readme specifies how to compile the

program Encoded data is sent to a binary file or a simple

text file (but isnrsquot ldquoreadablerdquo as text) Encoded data can only be ldquocheckedrdquo with a

decodercopy DJ Foreman 2012

16

Sample of prefix coding amp compression

copy DJ Foreman 2012

B e a r c a t s (8 bytes of text)010101100110001110001111 encoded as123456781234567812345678 3 bytes

A simple Huffman-like code(wo probability basis) 0 1

A 110

B 010

C 0001

E 10

R 011

S 111

T 001

Compression-Model types

Static Read all the data Compute frequencies Re-read the data amp encode

Dynamic Build simple basic model Modify model as more data is processed

copy DJ Foreman 2012 17

Standard Compressed-File Content

Three parts 24-byte header with magic (defines

codec) variable-length annotation block contiguous segment of audio data

Storage methodology network (big-endian) byte order multi-byte audio data may require byte

reversal in order to operate on it by the arithmetic unit of certain processors

copy DJ Foreman 2012 18

Example 1 Header file for AU files

copy DJ Foreman 2012 19

typedef unsigned long u_32 unsigned 32-bit integer

typedef struct

u_32 magic the ldquomagic numberrdquo

u_32 hdr_size byte offset to start of data

u_32 data_size length (optional)

u_32 encoding data encoding enumeration

u_32 sample_rate samples per second

u_32 channels of interleaved channels

Audio_filehdr

Following is from include ltmultimedialibaudiohgt

AUDIO_FILE_MAGIC ((u_32)0x2e736e64) ldquosndrdquo

Audio Encoding EnumerationsAUDIO_FILE_ENCODING_MULAW_8 (1) 8-bit ISDN u-law

AUDIO_FILE_ENCODING_LINEAR_8 (2) 8-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_16 (3) 16-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_32 (5) 32-bit linear PCM

AUDIO_FILE_ENCODING_FLOAT (6) 32-bit IEEE floating point

AUDIO_FILE_ENCODING_DOUBLE (7) 64-bit IEEE floating point

AUDIO_FILE_ENCODING_ADPCM_G721 (23) 4-bit CCITT g721 ADPCM

AUDIO_FILE_ENCODING_ADPCM_G723_3 (25) CCITT g723 3-bit ADPCM

AUDIO_FILE_ENCODING_ALAW_8 (27) 8-bit ISDN A-law

copy DJ Foreman 2012 20

ldquoLinearrdquo values are SIGNED intrsquosFloats are signed zero-centered normalized to ( -10 lt= x lt= 10 )

Example 2 ZIP files (PkWare)Purpose Size in bytes

Signature header 4

Required version 2

GP flags 2

Method 2

Last mod time 2

Last mod date 2

CRC-32 4

Compressed size 4

Uncompressed size 4

Filename length 2

Extra field length 2

File name variable

extra variable

copy DJ Foreman 2012 21

Ref httplivedocsadobecomflex3htmlhelphtmlcontent=ByteArrays_3html

More info on encoding httpwwwpkwarecomdocumentscase

studiesAPPNOTETXT

httpwwwfileinfocomfiletypescompressed

copy DJ Foreman 2012 22

Recording Bit-Rate

Constant (CBR) rate at which a codecs output data should be

consumed is constant Max bit-rate matters not the average Uses all available bandwidth Not good for storage (lossy)

Variable (VBR) Quantity of datatime unit (for output) varies Better quality for audio and video Slower to encode Supported by most portable devices post 2006

copy DJ Foreman 2012 23

Lab assignment ndash 1 ndash part 1

Write a program with 1 parameter If you are on an encoding team

Open a test file (eg ldquoxyztxtrdquo) Compress it using simple Huffman compression using

the frequency table from my FTP site Output compressed file (eg ldquoxyzencrdquo) to SAME folder

as input

If you are on a decoding team Open compressed file ldquoxyzencrdquo Decompress the input file Output file ldquoxyzdecrdquo so it can be compared to ldquoxyztxtrdquo

The parameter is the full input file path amp name (ie file names MUST NOT be hard-coded)

Remember a code can be a single bit

copy DJ Foreman 2012 24

25

Lab assignment 1 ndash part 2

Use ONLY MY TABLE for the frequencies Use ONLY uppercase letters of the table Do NOT use the special character

tables Find someone in class with the

opposite program and TEST your program to see if it properly encodes or decodes a message

Grading tree construction tree following proper binary output (encoders) proper character output (decoders)

copy DJ Foreman 2012

Working with ImagesModeling

Containers

A container is a FILE format NOT a code scheme

Eg AVI amp WAV are container formats Formats for storage Independent of encoding or content

Others Ogg ASF QuickTime RealMedia Matroska

DivX and MP4

copy DJ Foreman 2012 27

28

Movie (metadata)

Track

Media

Media Information

Container file format (MP4QT)

Chunk map

Comp time

Chunk offset

Key frame

Desc

Decode time

Priority map

Size map

Sample Table

Media Handler

Data information

Media header

Video header

Track header

Movie header

copy DJ Foreman 2012

Pixels A pixel is 3 color-dots (squares) in a collection of dots

(squares) making up the picture Each color-dot is for one of the three primary colors

RGB Minimum of 1 byte per color (depending on bit-depth

which varies with camera) thus at least 3 bytes per pixel

A 10MP camera (30 million color-dots) needs 30M bytes This gets compressed by the camera (JPEG format) Final file size = 30 million bytes divided by the amount

of compression (compression ratio) An 81 compression ratio would reduce that 30 million

bytes to 375 megabytes (as done on a 4 MP camera) A RAW file has NO compression

copy DJ Foreman 2012 29

Image creation

Basic mechanisms Photography

Film Digital ndashPhotoshop Gimp

Drawing Line BrushModeling

Do it yourself OpenGL (2x and 3x)

Modelingrendering programs Breeze designer Ayam 3DCanvas Makehuman Alice

copy DJ Foreman 2012 30

Photography (film) Silver halide crystals (AgCl AgBr AgI)

Exposure to light turns them black Developer removes the loose (exposed) halide leaving

pure silver (a negative image) Fixing bath flushes out unexposed AgHalide

Greatest desire of photographers - a medium with High speed (gathers light quickly) Low noise (no undesired artifacts) High resolution (very detailed images)

Visible lines per mm (lpm) at a specified contrast level

Notes Larger ldquoclumpsrdquo are faster but limit resolution Smaller ldquoclumpsrdquo are hard to coat evenly Randomness of coating prevents Moireacute patternshttpenwikipediaorgwikiMoirC3A9_pattern

copy DJ Foreman 2012 31

FYI ndash a little chemistry

Ag+ X- + hv rarr Ag+ + X + e-

1 Radiation frees the Br2 1 Ag ion + 1 electron (Ag+ + e- rarr Ag0) -gt 1 Ag atom3 Neighbors are now ldquosensitizedrdquo 4 Developing (Reducing) converts exposed AgX to

black metallic Ag and free X5 Remaining AgX and free X are washed away (after

mixing with thiosulfate to make it soluble)

This is one silver (Ag) ldquograinrdquo in the ldquolatent imagerdquo X is any halide (Br Cl etc)

copy DJ Foreman 2012 32

crystal

Radiation (light)

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Lossless Encoding

Huffman compression Bit string representing any symbol is never a

prefix of the bit string representing any other symbolie if 010 is the code for a symbol then no other symbol starts with 010

Frequency table must be known computable or included

Lossless ndash every character gets encoded Arithmetic encoding

Probabilistic algorithm Slightly superior to Huffman but often

patentedcopy DJ Foreman 2012 13

Methodology

Compute a model of the data eg a Huffman tree based on probability Probability determined from input file

Map the data according to the model Read 1 symbol Search the model (tree) for that symbol

If itrsquos a Huffman tree going left=0 right=1 Append each 0 or 1 to the code string When symbol is found you are done Note all symbols will have lt 8 bits (thus

compression)

copy DJ Foreman 2012 14

15

Huffman lab ndash more details Read write are always in bytes

You may need to pad You may have extraneous letters in the output

due to padding You donrsquot need to add a header to specify size

You have 2 weeks to do the lab Code may be in C C++ or Java The readme specifies how to compile the

program Encoded data is sent to a binary file or a simple

text file (but isnrsquot ldquoreadablerdquo as text) Encoded data can only be ldquocheckedrdquo with a

decodercopy DJ Foreman 2012

16

Sample of prefix coding amp compression

copy DJ Foreman 2012

B e a r c a t s (8 bytes of text)010101100110001110001111 encoded as123456781234567812345678 3 bytes

A simple Huffman-like code(wo probability basis) 0 1

A 110

B 010

C 0001

E 10

R 011

S 111

T 001

Compression-Model types

Static Read all the data Compute frequencies Re-read the data amp encode

Dynamic Build simple basic model Modify model as more data is processed

copy DJ Foreman 2012 17

Standard Compressed-File Content

Three parts 24-byte header with magic (defines

codec) variable-length annotation block contiguous segment of audio data

Storage methodology network (big-endian) byte order multi-byte audio data may require byte

reversal in order to operate on it by the arithmetic unit of certain processors

copy DJ Foreman 2012 18

Example 1 Header file for AU files

copy DJ Foreman 2012 19

typedef unsigned long u_32 unsigned 32-bit integer

typedef struct

u_32 magic the ldquomagic numberrdquo

u_32 hdr_size byte offset to start of data

u_32 data_size length (optional)

u_32 encoding data encoding enumeration

u_32 sample_rate samples per second

u_32 channels of interleaved channels

Audio_filehdr

Following is from include ltmultimedialibaudiohgt

AUDIO_FILE_MAGIC ((u_32)0x2e736e64) ldquosndrdquo

Audio Encoding EnumerationsAUDIO_FILE_ENCODING_MULAW_8 (1) 8-bit ISDN u-law

AUDIO_FILE_ENCODING_LINEAR_8 (2) 8-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_16 (3) 16-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_32 (5) 32-bit linear PCM

AUDIO_FILE_ENCODING_FLOAT (6) 32-bit IEEE floating point

AUDIO_FILE_ENCODING_DOUBLE (7) 64-bit IEEE floating point

AUDIO_FILE_ENCODING_ADPCM_G721 (23) 4-bit CCITT g721 ADPCM

AUDIO_FILE_ENCODING_ADPCM_G723_3 (25) CCITT g723 3-bit ADPCM

AUDIO_FILE_ENCODING_ALAW_8 (27) 8-bit ISDN A-law

copy DJ Foreman 2012 20

ldquoLinearrdquo values are SIGNED intrsquosFloats are signed zero-centered normalized to ( -10 lt= x lt= 10 )

Example 2 ZIP files (PkWare)Purpose Size in bytes

Signature header 4

Required version 2

GP flags 2

Method 2

Last mod time 2

Last mod date 2

CRC-32 4

Compressed size 4

Uncompressed size 4

Filename length 2

Extra field length 2

File name variable

extra variable

copy DJ Foreman 2012 21

Ref httplivedocsadobecomflex3htmlhelphtmlcontent=ByteArrays_3html

More info on encoding httpwwwpkwarecomdocumentscase

studiesAPPNOTETXT

httpwwwfileinfocomfiletypescompressed

copy DJ Foreman 2012 22

Recording Bit-Rate

Constant (CBR) rate at which a codecs output data should be

consumed is constant Max bit-rate matters not the average Uses all available bandwidth Not good for storage (lossy)

Variable (VBR) Quantity of datatime unit (for output) varies Better quality for audio and video Slower to encode Supported by most portable devices post 2006

copy DJ Foreman 2012 23

Lab assignment ndash 1 ndash part 1

Write a program with 1 parameter If you are on an encoding team

Open a test file (eg ldquoxyztxtrdquo) Compress it using simple Huffman compression using

the frequency table from my FTP site Output compressed file (eg ldquoxyzencrdquo) to SAME folder

as input

If you are on a decoding team Open compressed file ldquoxyzencrdquo Decompress the input file Output file ldquoxyzdecrdquo so it can be compared to ldquoxyztxtrdquo

The parameter is the full input file path amp name (ie file names MUST NOT be hard-coded)

Remember a code can be a single bit

copy DJ Foreman 2012 24

25

Lab assignment 1 ndash part 2

Use ONLY MY TABLE for the frequencies Use ONLY uppercase letters of the table Do NOT use the special character

tables Find someone in class with the

opposite program and TEST your program to see if it properly encodes or decodes a message

Grading tree construction tree following proper binary output (encoders) proper character output (decoders)

copy DJ Foreman 2012

Working with ImagesModeling

Containers

A container is a FILE format NOT a code scheme

Eg AVI amp WAV are container formats Formats for storage Independent of encoding or content

Others Ogg ASF QuickTime RealMedia Matroska

DivX and MP4

copy DJ Foreman 2012 27

28

Movie (metadata)

Track

Media

Media Information

Container file format (MP4QT)

Chunk map

Comp time

Chunk offset

Key frame

Desc

Decode time

Priority map

Size map

Sample Table

Media Handler

Data information

Media header

Video header

Track header

Movie header

copy DJ Foreman 2012

Pixels A pixel is 3 color-dots (squares) in a collection of dots

(squares) making up the picture Each color-dot is for one of the three primary colors

RGB Minimum of 1 byte per color (depending on bit-depth

which varies with camera) thus at least 3 bytes per pixel

A 10MP camera (30 million color-dots) needs 30M bytes This gets compressed by the camera (JPEG format) Final file size = 30 million bytes divided by the amount

of compression (compression ratio) An 81 compression ratio would reduce that 30 million

bytes to 375 megabytes (as done on a 4 MP camera) A RAW file has NO compression

copy DJ Foreman 2012 29

Image creation

Basic mechanisms Photography

Film Digital ndashPhotoshop Gimp

Drawing Line BrushModeling

Do it yourself OpenGL (2x and 3x)

Modelingrendering programs Breeze designer Ayam 3DCanvas Makehuman Alice

copy DJ Foreman 2012 30

Photography (film) Silver halide crystals (AgCl AgBr AgI)

Exposure to light turns them black Developer removes the loose (exposed) halide leaving

pure silver (a negative image) Fixing bath flushes out unexposed AgHalide

Greatest desire of photographers - a medium with High speed (gathers light quickly) Low noise (no undesired artifacts) High resolution (very detailed images)

Visible lines per mm (lpm) at a specified contrast level

Notes Larger ldquoclumpsrdquo are faster but limit resolution Smaller ldquoclumpsrdquo are hard to coat evenly Randomness of coating prevents Moireacute patternshttpenwikipediaorgwikiMoirC3A9_pattern

copy DJ Foreman 2012 31

FYI ndash a little chemistry

Ag+ X- + hv rarr Ag+ + X + e-

1 Radiation frees the Br2 1 Ag ion + 1 electron (Ag+ + e- rarr Ag0) -gt 1 Ag atom3 Neighbors are now ldquosensitizedrdquo 4 Developing (Reducing) converts exposed AgX to

black metallic Ag and free X5 Remaining AgX and free X are washed away (after

mixing with thiosulfate to make it soluble)

This is one silver (Ag) ldquograinrdquo in the ldquolatent imagerdquo X is any halide (Br Cl etc)

copy DJ Foreman 2012 32

crystal

Radiation (light)

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Methodology

Compute a model of the data eg a Huffman tree based on probability Probability determined from input file

Map the data according to the model Read 1 symbol Search the model (tree) for that symbol

If itrsquos a Huffman tree going left=0 right=1 Append each 0 or 1 to the code string When symbol is found you are done Note all symbols will have lt 8 bits (thus

compression)

copy DJ Foreman 2012 14

15

Huffman lab ndash more details Read write are always in bytes

You may need to pad You may have extraneous letters in the output

due to padding You donrsquot need to add a header to specify size

You have 2 weeks to do the lab Code may be in C C++ or Java The readme specifies how to compile the

program Encoded data is sent to a binary file or a simple

text file (but isnrsquot ldquoreadablerdquo as text) Encoded data can only be ldquocheckedrdquo with a

decodercopy DJ Foreman 2012

16

Sample of prefix coding amp compression

copy DJ Foreman 2012

B e a r c a t s (8 bytes of text)010101100110001110001111 encoded as123456781234567812345678 3 bytes

A simple Huffman-like code(wo probability basis) 0 1

A 110

B 010

C 0001

E 10

R 011

S 111

T 001

Compression-Model types

Static Read all the data Compute frequencies Re-read the data amp encode

Dynamic Build simple basic model Modify model as more data is processed

copy DJ Foreman 2012 17

Standard Compressed-File Content

Three parts 24-byte header with magic (defines

codec) variable-length annotation block contiguous segment of audio data

Storage methodology network (big-endian) byte order multi-byte audio data may require byte

reversal in order to operate on it by the arithmetic unit of certain processors

copy DJ Foreman 2012 18

Example 1 Header file for AU files

copy DJ Foreman 2012 19

typedef unsigned long u_32 unsigned 32-bit integer

typedef struct

u_32 magic the ldquomagic numberrdquo

u_32 hdr_size byte offset to start of data

u_32 data_size length (optional)

u_32 encoding data encoding enumeration

u_32 sample_rate samples per second

u_32 channels of interleaved channels

Audio_filehdr

Following is from include ltmultimedialibaudiohgt

AUDIO_FILE_MAGIC ((u_32)0x2e736e64) ldquosndrdquo

Audio Encoding EnumerationsAUDIO_FILE_ENCODING_MULAW_8 (1) 8-bit ISDN u-law

AUDIO_FILE_ENCODING_LINEAR_8 (2) 8-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_16 (3) 16-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_32 (5) 32-bit linear PCM

AUDIO_FILE_ENCODING_FLOAT (6) 32-bit IEEE floating point

AUDIO_FILE_ENCODING_DOUBLE (7) 64-bit IEEE floating point

AUDIO_FILE_ENCODING_ADPCM_G721 (23) 4-bit CCITT g721 ADPCM

AUDIO_FILE_ENCODING_ADPCM_G723_3 (25) CCITT g723 3-bit ADPCM

AUDIO_FILE_ENCODING_ALAW_8 (27) 8-bit ISDN A-law

copy DJ Foreman 2012 20

ldquoLinearrdquo values are SIGNED intrsquosFloats are signed zero-centered normalized to ( -10 lt= x lt= 10 )

Example 2 ZIP files (PkWare)Purpose Size in bytes

Signature header 4

Required version 2

GP flags 2

Method 2

Last mod time 2

Last mod date 2

CRC-32 4

Compressed size 4

Uncompressed size 4

Filename length 2

Extra field length 2

File name variable

extra variable

copy DJ Foreman 2012 21

Ref httplivedocsadobecomflex3htmlhelphtmlcontent=ByteArrays_3html

More info on encoding httpwwwpkwarecomdocumentscase

studiesAPPNOTETXT

httpwwwfileinfocomfiletypescompressed

copy DJ Foreman 2012 22

Recording Bit-Rate

Constant (CBR) rate at which a codecs output data should be

consumed is constant Max bit-rate matters not the average Uses all available bandwidth Not good for storage (lossy)

Variable (VBR) Quantity of datatime unit (for output) varies Better quality for audio and video Slower to encode Supported by most portable devices post 2006

copy DJ Foreman 2012 23

Lab assignment ndash 1 ndash part 1

Write a program with 1 parameter If you are on an encoding team

Open a test file (eg ldquoxyztxtrdquo) Compress it using simple Huffman compression using

the frequency table from my FTP site Output compressed file (eg ldquoxyzencrdquo) to SAME folder

as input

If you are on a decoding team Open compressed file ldquoxyzencrdquo Decompress the input file Output file ldquoxyzdecrdquo so it can be compared to ldquoxyztxtrdquo

The parameter is the full input file path amp name (ie file names MUST NOT be hard-coded)

Remember a code can be a single bit

copy DJ Foreman 2012 24

25

Lab assignment 1 ndash part 2

Use ONLY MY TABLE for the frequencies Use ONLY uppercase letters of the table Do NOT use the special character

tables Find someone in class with the

opposite program and TEST your program to see if it properly encodes or decodes a message

Grading tree construction tree following proper binary output (encoders) proper character output (decoders)

copy DJ Foreman 2012

Working with ImagesModeling

Containers

A container is a FILE format NOT a code scheme

Eg AVI amp WAV are container formats Formats for storage Independent of encoding or content

Others Ogg ASF QuickTime RealMedia Matroska

DivX and MP4

copy DJ Foreman 2012 27

28

Movie (metadata)

Track

Media

Media Information

Container file format (MP4QT)

Chunk map

Comp time

Chunk offset

Key frame

Desc

Decode time

Priority map

Size map

Sample Table

Media Handler

Data information

Media header

Video header

Track header

Movie header

copy DJ Foreman 2012

Pixels A pixel is 3 color-dots (squares) in a collection of dots

(squares) making up the picture Each color-dot is for one of the three primary colors

RGB Minimum of 1 byte per color (depending on bit-depth

which varies with camera) thus at least 3 bytes per pixel

A 10MP camera (30 million color-dots) needs 30M bytes This gets compressed by the camera (JPEG format) Final file size = 30 million bytes divided by the amount

of compression (compression ratio) An 81 compression ratio would reduce that 30 million

bytes to 375 megabytes (as done on a 4 MP camera) A RAW file has NO compression

copy DJ Foreman 2012 29

Image creation

Basic mechanisms Photography

Film Digital ndashPhotoshop Gimp

Drawing Line BrushModeling

Do it yourself OpenGL (2x and 3x)

Modelingrendering programs Breeze designer Ayam 3DCanvas Makehuman Alice

copy DJ Foreman 2012 30

Photography (film) Silver halide crystals (AgCl AgBr AgI)

Exposure to light turns them black Developer removes the loose (exposed) halide leaving

pure silver (a negative image) Fixing bath flushes out unexposed AgHalide

Greatest desire of photographers - a medium with High speed (gathers light quickly) Low noise (no undesired artifacts) High resolution (very detailed images)

Visible lines per mm (lpm) at a specified contrast level

Notes Larger ldquoclumpsrdquo are faster but limit resolution Smaller ldquoclumpsrdquo are hard to coat evenly Randomness of coating prevents Moireacute patternshttpenwikipediaorgwikiMoirC3A9_pattern

copy DJ Foreman 2012 31

FYI ndash a little chemistry

Ag+ X- + hv rarr Ag+ + X + e-

1 Radiation frees the Br2 1 Ag ion + 1 electron (Ag+ + e- rarr Ag0) -gt 1 Ag atom3 Neighbors are now ldquosensitizedrdquo 4 Developing (Reducing) converts exposed AgX to

black metallic Ag and free X5 Remaining AgX and free X are washed away (after

mixing with thiosulfate to make it soluble)

This is one silver (Ag) ldquograinrdquo in the ldquolatent imagerdquo X is any halide (Br Cl etc)

copy DJ Foreman 2012 32

crystal

Radiation (light)

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

15

Huffman lab ndash more details Read write are always in bytes

You may need to pad You may have extraneous letters in the output

due to padding You donrsquot need to add a header to specify size

You have 2 weeks to do the lab Code may be in C C++ or Java The readme specifies how to compile the

program Encoded data is sent to a binary file or a simple

text file (but isnrsquot ldquoreadablerdquo as text) Encoded data can only be ldquocheckedrdquo with a

decodercopy DJ Foreman 2012

16

Sample of prefix coding amp compression

copy DJ Foreman 2012

B e a r c a t s (8 bytes of text)010101100110001110001111 encoded as123456781234567812345678 3 bytes

A simple Huffman-like code(wo probability basis) 0 1

A 110

B 010

C 0001

E 10

R 011

S 111

T 001

Compression-Model types

Static Read all the data Compute frequencies Re-read the data amp encode

Dynamic Build simple basic model Modify model as more data is processed

copy DJ Foreman 2012 17

Standard Compressed-File Content

Three parts 24-byte header with magic (defines

codec) variable-length annotation block contiguous segment of audio data

Storage methodology network (big-endian) byte order multi-byte audio data may require byte

reversal in order to operate on it by the arithmetic unit of certain processors

copy DJ Foreman 2012 18

Example 1 Header file for AU files

copy DJ Foreman 2012 19

typedef unsigned long u_32 unsigned 32-bit integer

typedef struct

u_32 magic the ldquomagic numberrdquo

u_32 hdr_size byte offset to start of data

u_32 data_size length (optional)

u_32 encoding data encoding enumeration

u_32 sample_rate samples per second

u_32 channels of interleaved channels

Audio_filehdr

Following is from include ltmultimedialibaudiohgt

AUDIO_FILE_MAGIC ((u_32)0x2e736e64) ldquosndrdquo

Audio Encoding EnumerationsAUDIO_FILE_ENCODING_MULAW_8 (1) 8-bit ISDN u-law

AUDIO_FILE_ENCODING_LINEAR_8 (2) 8-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_16 (3) 16-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_32 (5) 32-bit linear PCM

AUDIO_FILE_ENCODING_FLOAT (6) 32-bit IEEE floating point

AUDIO_FILE_ENCODING_DOUBLE (7) 64-bit IEEE floating point

AUDIO_FILE_ENCODING_ADPCM_G721 (23) 4-bit CCITT g721 ADPCM

AUDIO_FILE_ENCODING_ADPCM_G723_3 (25) CCITT g723 3-bit ADPCM

AUDIO_FILE_ENCODING_ALAW_8 (27) 8-bit ISDN A-law

copy DJ Foreman 2012 20

ldquoLinearrdquo values are SIGNED intrsquosFloats are signed zero-centered normalized to ( -10 lt= x lt= 10 )

Example 2 ZIP files (PkWare)Purpose Size in bytes

Signature header 4

Required version 2

GP flags 2

Method 2

Last mod time 2

Last mod date 2

CRC-32 4

Compressed size 4

Uncompressed size 4

Filename length 2

Extra field length 2

File name variable

extra variable

copy DJ Foreman 2012 21

Ref httplivedocsadobecomflex3htmlhelphtmlcontent=ByteArrays_3html

More info on encoding httpwwwpkwarecomdocumentscase

studiesAPPNOTETXT

httpwwwfileinfocomfiletypescompressed

copy DJ Foreman 2012 22

Recording Bit-Rate

Constant (CBR) rate at which a codecs output data should be

consumed is constant Max bit-rate matters not the average Uses all available bandwidth Not good for storage (lossy)

Variable (VBR) Quantity of datatime unit (for output) varies Better quality for audio and video Slower to encode Supported by most portable devices post 2006

copy DJ Foreman 2012 23

Lab assignment ndash 1 ndash part 1

Write a program with 1 parameter If you are on an encoding team

Open a test file (eg ldquoxyztxtrdquo) Compress it using simple Huffman compression using

the frequency table from my FTP site Output compressed file (eg ldquoxyzencrdquo) to SAME folder

as input

If you are on a decoding team Open compressed file ldquoxyzencrdquo Decompress the input file Output file ldquoxyzdecrdquo so it can be compared to ldquoxyztxtrdquo

The parameter is the full input file path amp name (ie file names MUST NOT be hard-coded)

Remember a code can be a single bit

copy DJ Foreman 2012 24

25

Lab assignment 1 ndash part 2

Use ONLY MY TABLE for the frequencies Use ONLY uppercase letters of the table Do NOT use the special character

tables Find someone in class with the

opposite program and TEST your program to see if it properly encodes or decodes a message

Grading tree construction tree following proper binary output (encoders) proper character output (decoders)

copy DJ Foreman 2012

Working with ImagesModeling

Containers

A container is a FILE format NOT a code scheme

Eg AVI amp WAV are container formats Formats for storage Independent of encoding or content

Others Ogg ASF QuickTime RealMedia Matroska

DivX and MP4

copy DJ Foreman 2012 27

28

Movie (metadata)

Track

Media

Media Information

Container file format (MP4QT)

Chunk map

Comp time

Chunk offset

Key frame

Desc

Decode time

Priority map

Size map

Sample Table

Media Handler

Data information

Media header

Video header

Track header

Movie header

copy DJ Foreman 2012

Pixels A pixel is 3 color-dots (squares) in a collection of dots

(squares) making up the picture Each color-dot is for one of the three primary colors

RGB Minimum of 1 byte per color (depending on bit-depth

which varies with camera) thus at least 3 bytes per pixel

A 10MP camera (30 million color-dots) needs 30M bytes This gets compressed by the camera (JPEG format) Final file size = 30 million bytes divided by the amount

of compression (compression ratio) An 81 compression ratio would reduce that 30 million

bytes to 375 megabytes (as done on a 4 MP camera) A RAW file has NO compression

copy DJ Foreman 2012 29

Image creation

Basic mechanisms Photography

Film Digital ndashPhotoshop Gimp

Drawing Line BrushModeling

Do it yourself OpenGL (2x and 3x)

Modelingrendering programs Breeze designer Ayam 3DCanvas Makehuman Alice

copy DJ Foreman 2012 30

Photography (film) Silver halide crystals (AgCl AgBr AgI)

Exposure to light turns them black Developer removes the loose (exposed) halide leaving

pure silver (a negative image) Fixing bath flushes out unexposed AgHalide

Greatest desire of photographers - a medium with High speed (gathers light quickly) Low noise (no undesired artifacts) High resolution (very detailed images)

Visible lines per mm (lpm) at a specified contrast level

Notes Larger ldquoclumpsrdquo are faster but limit resolution Smaller ldquoclumpsrdquo are hard to coat evenly Randomness of coating prevents Moireacute patternshttpenwikipediaorgwikiMoirC3A9_pattern

copy DJ Foreman 2012 31

FYI ndash a little chemistry

Ag+ X- + hv rarr Ag+ + X + e-

1 Radiation frees the Br2 1 Ag ion + 1 electron (Ag+ + e- rarr Ag0) -gt 1 Ag atom3 Neighbors are now ldquosensitizedrdquo 4 Developing (Reducing) converts exposed AgX to

black metallic Ag and free X5 Remaining AgX and free X are washed away (after

mixing with thiosulfate to make it soluble)

This is one silver (Ag) ldquograinrdquo in the ldquolatent imagerdquo X is any halide (Br Cl etc)

copy DJ Foreman 2012 32

crystal

Radiation (light)

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

16

Sample of prefix coding amp compression

copy DJ Foreman 2012

B e a r c a t s (8 bytes of text)010101100110001110001111 encoded as123456781234567812345678 3 bytes

A simple Huffman-like code(wo probability basis) 0 1

A 110

B 010

C 0001

E 10

R 011

S 111

T 001

Compression-Model types

Static Read all the data Compute frequencies Re-read the data amp encode

Dynamic Build simple basic model Modify model as more data is processed

copy DJ Foreman 2012 17

Standard Compressed-File Content

Three parts 24-byte header with magic (defines

codec) variable-length annotation block contiguous segment of audio data

Storage methodology network (big-endian) byte order multi-byte audio data may require byte

reversal in order to operate on it by the arithmetic unit of certain processors

copy DJ Foreman 2012 18

Example 1 Header file for AU files

copy DJ Foreman 2012 19

typedef unsigned long u_32 unsigned 32-bit integer

typedef struct

u_32 magic the ldquomagic numberrdquo

u_32 hdr_size byte offset to start of data

u_32 data_size length (optional)

u_32 encoding data encoding enumeration

u_32 sample_rate samples per second

u_32 channels of interleaved channels

Audio_filehdr

Following is from include ltmultimedialibaudiohgt

AUDIO_FILE_MAGIC ((u_32)0x2e736e64) ldquosndrdquo

Audio Encoding EnumerationsAUDIO_FILE_ENCODING_MULAW_8 (1) 8-bit ISDN u-law

AUDIO_FILE_ENCODING_LINEAR_8 (2) 8-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_16 (3) 16-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_32 (5) 32-bit linear PCM

AUDIO_FILE_ENCODING_FLOAT (6) 32-bit IEEE floating point

AUDIO_FILE_ENCODING_DOUBLE (7) 64-bit IEEE floating point

AUDIO_FILE_ENCODING_ADPCM_G721 (23) 4-bit CCITT g721 ADPCM

AUDIO_FILE_ENCODING_ADPCM_G723_3 (25) CCITT g723 3-bit ADPCM

AUDIO_FILE_ENCODING_ALAW_8 (27) 8-bit ISDN A-law

copy DJ Foreman 2012 20

ldquoLinearrdquo values are SIGNED intrsquosFloats are signed zero-centered normalized to ( -10 lt= x lt= 10 )

Example 2 ZIP files (PkWare)Purpose Size in bytes

Signature header 4

Required version 2

GP flags 2

Method 2

Last mod time 2

Last mod date 2

CRC-32 4

Compressed size 4

Uncompressed size 4

Filename length 2

Extra field length 2

File name variable

extra variable

copy DJ Foreman 2012 21

Ref httplivedocsadobecomflex3htmlhelphtmlcontent=ByteArrays_3html

More info on encoding httpwwwpkwarecomdocumentscase

studiesAPPNOTETXT

httpwwwfileinfocomfiletypescompressed

copy DJ Foreman 2012 22

Recording Bit-Rate

Constant (CBR) rate at which a codecs output data should be

consumed is constant Max bit-rate matters not the average Uses all available bandwidth Not good for storage (lossy)

Variable (VBR) Quantity of datatime unit (for output) varies Better quality for audio and video Slower to encode Supported by most portable devices post 2006

copy DJ Foreman 2012 23

Lab assignment ndash 1 ndash part 1

Write a program with 1 parameter If you are on an encoding team

Open a test file (eg ldquoxyztxtrdquo) Compress it using simple Huffman compression using

the frequency table from my FTP site Output compressed file (eg ldquoxyzencrdquo) to SAME folder

as input

If you are on a decoding team Open compressed file ldquoxyzencrdquo Decompress the input file Output file ldquoxyzdecrdquo so it can be compared to ldquoxyztxtrdquo

The parameter is the full input file path amp name (ie file names MUST NOT be hard-coded)

Remember a code can be a single bit

copy DJ Foreman 2012 24

25

Lab assignment 1 ndash part 2

Use ONLY MY TABLE for the frequencies Use ONLY uppercase letters of the table Do NOT use the special character

tables Find someone in class with the

opposite program and TEST your program to see if it properly encodes or decodes a message

Grading tree construction tree following proper binary output (encoders) proper character output (decoders)

copy DJ Foreman 2012

Working with ImagesModeling

Containers

A container is a FILE format NOT a code scheme

Eg AVI amp WAV are container formats Formats for storage Independent of encoding or content

Others Ogg ASF QuickTime RealMedia Matroska

DivX and MP4

copy DJ Foreman 2012 27

28

Movie (metadata)

Track

Media

Media Information

Container file format (MP4QT)

Chunk map

Comp time

Chunk offset

Key frame

Desc

Decode time

Priority map

Size map

Sample Table

Media Handler

Data information

Media header

Video header

Track header

Movie header

copy DJ Foreman 2012

Pixels A pixel is 3 color-dots (squares) in a collection of dots

(squares) making up the picture Each color-dot is for one of the three primary colors

RGB Minimum of 1 byte per color (depending on bit-depth

which varies with camera) thus at least 3 bytes per pixel

A 10MP camera (30 million color-dots) needs 30M bytes This gets compressed by the camera (JPEG format) Final file size = 30 million bytes divided by the amount

of compression (compression ratio) An 81 compression ratio would reduce that 30 million

bytes to 375 megabytes (as done on a 4 MP camera) A RAW file has NO compression

copy DJ Foreman 2012 29

Image creation

Basic mechanisms Photography

Film Digital ndashPhotoshop Gimp

Drawing Line BrushModeling

Do it yourself OpenGL (2x and 3x)

Modelingrendering programs Breeze designer Ayam 3DCanvas Makehuman Alice

copy DJ Foreman 2012 30

Photography (film) Silver halide crystals (AgCl AgBr AgI)

Exposure to light turns them black Developer removes the loose (exposed) halide leaving

pure silver (a negative image) Fixing bath flushes out unexposed AgHalide

Greatest desire of photographers - a medium with High speed (gathers light quickly) Low noise (no undesired artifacts) High resolution (very detailed images)

Visible lines per mm (lpm) at a specified contrast level

Notes Larger ldquoclumpsrdquo are faster but limit resolution Smaller ldquoclumpsrdquo are hard to coat evenly Randomness of coating prevents Moireacute patternshttpenwikipediaorgwikiMoirC3A9_pattern

copy DJ Foreman 2012 31

FYI ndash a little chemistry

Ag+ X- + hv rarr Ag+ + X + e-

1 Radiation frees the Br2 1 Ag ion + 1 electron (Ag+ + e- rarr Ag0) -gt 1 Ag atom3 Neighbors are now ldquosensitizedrdquo 4 Developing (Reducing) converts exposed AgX to

black metallic Ag and free X5 Remaining AgX and free X are washed away (after

mixing with thiosulfate to make it soluble)

This is one silver (Ag) ldquograinrdquo in the ldquolatent imagerdquo X is any halide (Br Cl etc)

copy DJ Foreman 2012 32

crystal

Radiation (light)

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Compression-Model types

Static Read all the data Compute frequencies Re-read the data amp encode

Dynamic Build simple basic model Modify model as more data is processed

copy DJ Foreman 2012 17

Standard Compressed-File Content

Three parts 24-byte header with magic (defines

codec) variable-length annotation block contiguous segment of audio data

Storage methodology network (big-endian) byte order multi-byte audio data may require byte

reversal in order to operate on it by the arithmetic unit of certain processors

copy DJ Foreman 2012 18

Example 1 Header file for AU files

copy DJ Foreman 2012 19

typedef unsigned long u_32 unsigned 32-bit integer

typedef struct

u_32 magic the ldquomagic numberrdquo

u_32 hdr_size byte offset to start of data

u_32 data_size length (optional)

u_32 encoding data encoding enumeration

u_32 sample_rate samples per second

u_32 channels of interleaved channels

Audio_filehdr

Following is from include ltmultimedialibaudiohgt

AUDIO_FILE_MAGIC ((u_32)0x2e736e64) ldquosndrdquo

Audio Encoding EnumerationsAUDIO_FILE_ENCODING_MULAW_8 (1) 8-bit ISDN u-law

AUDIO_FILE_ENCODING_LINEAR_8 (2) 8-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_16 (3) 16-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_32 (5) 32-bit linear PCM

AUDIO_FILE_ENCODING_FLOAT (6) 32-bit IEEE floating point

AUDIO_FILE_ENCODING_DOUBLE (7) 64-bit IEEE floating point

AUDIO_FILE_ENCODING_ADPCM_G721 (23) 4-bit CCITT g721 ADPCM

AUDIO_FILE_ENCODING_ADPCM_G723_3 (25) CCITT g723 3-bit ADPCM

AUDIO_FILE_ENCODING_ALAW_8 (27) 8-bit ISDN A-law

copy DJ Foreman 2012 20

ldquoLinearrdquo values are SIGNED intrsquosFloats are signed zero-centered normalized to ( -10 lt= x lt= 10 )

Example 2 ZIP files (PkWare)Purpose Size in bytes

Signature header 4

Required version 2

GP flags 2

Method 2

Last mod time 2

Last mod date 2

CRC-32 4

Compressed size 4

Uncompressed size 4

Filename length 2

Extra field length 2

File name variable

extra variable

copy DJ Foreman 2012 21

Ref httplivedocsadobecomflex3htmlhelphtmlcontent=ByteArrays_3html

More info on encoding httpwwwpkwarecomdocumentscase

studiesAPPNOTETXT

httpwwwfileinfocomfiletypescompressed

copy DJ Foreman 2012 22

Recording Bit-Rate

Constant (CBR) rate at which a codecs output data should be

consumed is constant Max bit-rate matters not the average Uses all available bandwidth Not good for storage (lossy)

Variable (VBR) Quantity of datatime unit (for output) varies Better quality for audio and video Slower to encode Supported by most portable devices post 2006

copy DJ Foreman 2012 23

Lab assignment ndash 1 ndash part 1

Write a program with 1 parameter If you are on an encoding team

Open a test file (eg ldquoxyztxtrdquo) Compress it using simple Huffman compression using

the frequency table from my FTP site Output compressed file (eg ldquoxyzencrdquo) to SAME folder

as input

If you are on a decoding team Open compressed file ldquoxyzencrdquo Decompress the input file Output file ldquoxyzdecrdquo so it can be compared to ldquoxyztxtrdquo

The parameter is the full input file path amp name (ie file names MUST NOT be hard-coded)

Remember a code can be a single bit

copy DJ Foreman 2012 24

25

Lab assignment 1 ndash part 2

Use ONLY MY TABLE for the frequencies Use ONLY uppercase letters of the table Do NOT use the special character

tables Find someone in class with the

opposite program and TEST your program to see if it properly encodes or decodes a message

Grading tree construction tree following proper binary output (encoders) proper character output (decoders)

copy DJ Foreman 2012

Working with ImagesModeling

Containers

A container is a FILE format NOT a code scheme

Eg AVI amp WAV are container formats Formats for storage Independent of encoding or content

Others Ogg ASF QuickTime RealMedia Matroska

DivX and MP4

copy DJ Foreman 2012 27

28

Movie (metadata)

Track

Media

Media Information

Container file format (MP4QT)

Chunk map

Comp time

Chunk offset

Key frame

Desc

Decode time

Priority map

Size map

Sample Table

Media Handler

Data information

Media header

Video header

Track header

Movie header

copy DJ Foreman 2012

Pixels A pixel is 3 color-dots (squares) in a collection of dots

(squares) making up the picture Each color-dot is for one of the three primary colors

RGB Minimum of 1 byte per color (depending on bit-depth

which varies with camera) thus at least 3 bytes per pixel

A 10MP camera (30 million color-dots) needs 30M bytes This gets compressed by the camera (JPEG format) Final file size = 30 million bytes divided by the amount

of compression (compression ratio) An 81 compression ratio would reduce that 30 million

bytes to 375 megabytes (as done on a 4 MP camera) A RAW file has NO compression

copy DJ Foreman 2012 29

Image creation

Basic mechanisms Photography

Film Digital ndashPhotoshop Gimp

Drawing Line BrushModeling

Do it yourself OpenGL (2x and 3x)

Modelingrendering programs Breeze designer Ayam 3DCanvas Makehuman Alice

copy DJ Foreman 2012 30

Photography (film) Silver halide crystals (AgCl AgBr AgI)

Exposure to light turns them black Developer removes the loose (exposed) halide leaving

pure silver (a negative image) Fixing bath flushes out unexposed AgHalide

Greatest desire of photographers - a medium with High speed (gathers light quickly) Low noise (no undesired artifacts) High resolution (very detailed images)

Visible lines per mm (lpm) at a specified contrast level

Notes Larger ldquoclumpsrdquo are faster but limit resolution Smaller ldquoclumpsrdquo are hard to coat evenly Randomness of coating prevents Moireacute patternshttpenwikipediaorgwikiMoirC3A9_pattern

copy DJ Foreman 2012 31

FYI ndash a little chemistry

Ag+ X- + hv rarr Ag+ + X + e-

1 Radiation frees the Br2 1 Ag ion + 1 electron (Ag+ + e- rarr Ag0) -gt 1 Ag atom3 Neighbors are now ldquosensitizedrdquo 4 Developing (Reducing) converts exposed AgX to

black metallic Ag and free X5 Remaining AgX and free X are washed away (after

mixing with thiosulfate to make it soluble)

This is one silver (Ag) ldquograinrdquo in the ldquolatent imagerdquo X is any halide (Br Cl etc)

copy DJ Foreman 2012 32

crystal

Radiation (light)

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Standard Compressed-File Content

Three parts 24-byte header with magic (defines

codec) variable-length annotation block contiguous segment of audio data

Storage methodology network (big-endian) byte order multi-byte audio data may require byte

reversal in order to operate on it by the arithmetic unit of certain processors

copy DJ Foreman 2012 18

Example 1 Header file for AU files

copy DJ Foreman 2012 19

typedef unsigned long u_32 unsigned 32-bit integer

typedef struct

u_32 magic the ldquomagic numberrdquo

u_32 hdr_size byte offset to start of data

u_32 data_size length (optional)

u_32 encoding data encoding enumeration

u_32 sample_rate samples per second

u_32 channels of interleaved channels

Audio_filehdr

Following is from include ltmultimedialibaudiohgt

AUDIO_FILE_MAGIC ((u_32)0x2e736e64) ldquosndrdquo

Audio Encoding EnumerationsAUDIO_FILE_ENCODING_MULAW_8 (1) 8-bit ISDN u-law

AUDIO_FILE_ENCODING_LINEAR_8 (2) 8-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_16 (3) 16-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_32 (5) 32-bit linear PCM

AUDIO_FILE_ENCODING_FLOAT (6) 32-bit IEEE floating point

AUDIO_FILE_ENCODING_DOUBLE (7) 64-bit IEEE floating point

AUDIO_FILE_ENCODING_ADPCM_G721 (23) 4-bit CCITT g721 ADPCM

AUDIO_FILE_ENCODING_ADPCM_G723_3 (25) CCITT g723 3-bit ADPCM

AUDIO_FILE_ENCODING_ALAW_8 (27) 8-bit ISDN A-law

copy DJ Foreman 2012 20

ldquoLinearrdquo values are SIGNED intrsquosFloats are signed zero-centered normalized to ( -10 lt= x lt= 10 )

Example 2 ZIP files (PkWare)Purpose Size in bytes

Signature header 4

Required version 2

GP flags 2

Method 2

Last mod time 2

Last mod date 2

CRC-32 4

Compressed size 4

Uncompressed size 4

Filename length 2

Extra field length 2

File name variable

extra variable

copy DJ Foreman 2012 21

Ref httplivedocsadobecomflex3htmlhelphtmlcontent=ByteArrays_3html

More info on encoding httpwwwpkwarecomdocumentscase

studiesAPPNOTETXT

httpwwwfileinfocomfiletypescompressed

copy DJ Foreman 2012 22

Recording Bit-Rate

Constant (CBR) rate at which a codecs output data should be

consumed is constant Max bit-rate matters not the average Uses all available bandwidth Not good for storage (lossy)

Variable (VBR) Quantity of datatime unit (for output) varies Better quality for audio and video Slower to encode Supported by most portable devices post 2006

copy DJ Foreman 2012 23

Lab assignment ndash 1 ndash part 1

Write a program with 1 parameter If you are on an encoding team

Open a test file (eg ldquoxyztxtrdquo) Compress it using simple Huffman compression using

the frequency table from my FTP site Output compressed file (eg ldquoxyzencrdquo) to SAME folder

as input

If you are on a decoding team Open compressed file ldquoxyzencrdquo Decompress the input file Output file ldquoxyzdecrdquo so it can be compared to ldquoxyztxtrdquo

The parameter is the full input file path amp name (ie file names MUST NOT be hard-coded)

Remember a code can be a single bit

copy DJ Foreman 2012 24

25

Lab assignment 1 ndash part 2

Use ONLY MY TABLE for the frequencies Use ONLY uppercase letters of the table Do NOT use the special character

tables Find someone in class with the

opposite program and TEST your program to see if it properly encodes or decodes a message

Grading tree construction tree following proper binary output (encoders) proper character output (decoders)

copy DJ Foreman 2012

Working with ImagesModeling

Containers

A container is a FILE format NOT a code scheme

Eg AVI amp WAV are container formats Formats for storage Independent of encoding or content

Others Ogg ASF QuickTime RealMedia Matroska

DivX and MP4

copy DJ Foreman 2012 27

28

Movie (metadata)

Track

Media

Media Information

Container file format (MP4QT)

Chunk map

Comp time

Chunk offset

Key frame

Desc

Decode time

Priority map

Size map

Sample Table

Media Handler

Data information

Media header

Video header

Track header

Movie header

copy DJ Foreman 2012

Pixels A pixel is 3 color-dots (squares) in a collection of dots

(squares) making up the picture Each color-dot is for one of the three primary colors

RGB Minimum of 1 byte per color (depending on bit-depth

which varies with camera) thus at least 3 bytes per pixel

A 10MP camera (30 million color-dots) needs 30M bytes This gets compressed by the camera (JPEG format) Final file size = 30 million bytes divided by the amount

of compression (compression ratio) An 81 compression ratio would reduce that 30 million

bytes to 375 megabytes (as done on a 4 MP camera) A RAW file has NO compression

copy DJ Foreman 2012 29

Image creation

Basic mechanisms Photography

Film Digital ndashPhotoshop Gimp

Drawing Line BrushModeling

Do it yourself OpenGL (2x and 3x)

Modelingrendering programs Breeze designer Ayam 3DCanvas Makehuman Alice

copy DJ Foreman 2012 30

Photography (film) Silver halide crystals (AgCl AgBr AgI)

Exposure to light turns them black Developer removes the loose (exposed) halide leaving

pure silver (a negative image) Fixing bath flushes out unexposed AgHalide

Greatest desire of photographers - a medium with High speed (gathers light quickly) Low noise (no undesired artifacts) High resolution (very detailed images)

Visible lines per mm (lpm) at a specified contrast level

Notes Larger ldquoclumpsrdquo are faster but limit resolution Smaller ldquoclumpsrdquo are hard to coat evenly Randomness of coating prevents Moireacute patternshttpenwikipediaorgwikiMoirC3A9_pattern

copy DJ Foreman 2012 31

FYI ndash a little chemistry

Ag+ X- + hv rarr Ag+ + X + e-

1 Radiation frees the Br2 1 Ag ion + 1 electron (Ag+ + e- rarr Ag0) -gt 1 Ag atom3 Neighbors are now ldquosensitizedrdquo 4 Developing (Reducing) converts exposed AgX to

black metallic Ag and free X5 Remaining AgX and free X are washed away (after

mixing with thiosulfate to make it soluble)

This is one silver (Ag) ldquograinrdquo in the ldquolatent imagerdquo X is any halide (Br Cl etc)

copy DJ Foreman 2012 32

crystal

Radiation (light)

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Example 1 Header file for AU files

copy DJ Foreman 2012 19

typedef unsigned long u_32 unsigned 32-bit integer

typedef struct

u_32 magic the ldquomagic numberrdquo

u_32 hdr_size byte offset to start of data

u_32 data_size length (optional)

u_32 encoding data encoding enumeration

u_32 sample_rate samples per second

u_32 channels of interleaved channels

Audio_filehdr

Following is from include ltmultimedialibaudiohgt

AUDIO_FILE_MAGIC ((u_32)0x2e736e64) ldquosndrdquo

Audio Encoding EnumerationsAUDIO_FILE_ENCODING_MULAW_8 (1) 8-bit ISDN u-law

AUDIO_FILE_ENCODING_LINEAR_8 (2) 8-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_16 (3) 16-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_32 (5) 32-bit linear PCM

AUDIO_FILE_ENCODING_FLOAT (6) 32-bit IEEE floating point

AUDIO_FILE_ENCODING_DOUBLE (7) 64-bit IEEE floating point

AUDIO_FILE_ENCODING_ADPCM_G721 (23) 4-bit CCITT g721 ADPCM

AUDIO_FILE_ENCODING_ADPCM_G723_3 (25) CCITT g723 3-bit ADPCM

AUDIO_FILE_ENCODING_ALAW_8 (27) 8-bit ISDN A-law

copy DJ Foreman 2012 20

ldquoLinearrdquo values are SIGNED intrsquosFloats are signed zero-centered normalized to ( -10 lt= x lt= 10 )

Example 2 ZIP files (PkWare)Purpose Size in bytes

Signature header 4

Required version 2

GP flags 2

Method 2

Last mod time 2

Last mod date 2

CRC-32 4

Compressed size 4

Uncompressed size 4

Filename length 2

Extra field length 2

File name variable

extra variable

copy DJ Foreman 2012 21

Ref httplivedocsadobecomflex3htmlhelphtmlcontent=ByteArrays_3html

More info on encoding httpwwwpkwarecomdocumentscase

studiesAPPNOTETXT

httpwwwfileinfocomfiletypescompressed

copy DJ Foreman 2012 22

Recording Bit-Rate

Constant (CBR) rate at which a codecs output data should be

consumed is constant Max bit-rate matters not the average Uses all available bandwidth Not good for storage (lossy)

Variable (VBR) Quantity of datatime unit (for output) varies Better quality for audio and video Slower to encode Supported by most portable devices post 2006

copy DJ Foreman 2012 23

Lab assignment ndash 1 ndash part 1

Write a program with 1 parameter If you are on an encoding team

Open a test file (eg ldquoxyztxtrdquo) Compress it using simple Huffman compression using

the frequency table from my FTP site Output compressed file (eg ldquoxyzencrdquo) to SAME folder

as input

If you are on a decoding team Open compressed file ldquoxyzencrdquo Decompress the input file Output file ldquoxyzdecrdquo so it can be compared to ldquoxyztxtrdquo

The parameter is the full input file path amp name (ie file names MUST NOT be hard-coded)

Remember a code can be a single bit

copy DJ Foreman 2012 24

25

Lab assignment 1 ndash part 2

Use ONLY MY TABLE for the frequencies Use ONLY uppercase letters of the table Do NOT use the special character

tables Find someone in class with the

opposite program and TEST your program to see if it properly encodes or decodes a message

Grading tree construction tree following proper binary output (encoders) proper character output (decoders)

copy DJ Foreman 2012

Working with ImagesModeling

Containers

A container is a FILE format NOT a code scheme

Eg AVI amp WAV are container formats Formats for storage Independent of encoding or content

Others Ogg ASF QuickTime RealMedia Matroska

DivX and MP4

copy DJ Foreman 2012 27

28

Movie (metadata)

Track

Media

Media Information

Container file format (MP4QT)

Chunk map

Comp time

Chunk offset

Key frame

Desc

Decode time

Priority map

Size map

Sample Table

Media Handler

Data information

Media header

Video header

Track header

Movie header

copy DJ Foreman 2012

Pixels A pixel is 3 color-dots (squares) in a collection of dots

(squares) making up the picture Each color-dot is for one of the three primary colors

RGB Minimum of 1 byte per color (depending on bit-depth

which varies with camera) thus at least 3 bytes per pixel

A 10MP camera (30 million color-dots) needs 30M bytes This gets compressed by the camera (JPEG format) Final file size = 30 million bytes divided by the amount

of compression (compression ratio) An 81 compression ratio would reduce that 30 million

bytes to 375 megabytes (as done on a 4 MP camera) A RAW file has NO compression

copy DJ Foreman 2012 29

Image creation

Basic mechanisms Photography

Film Digital ndashPhotoshop Gimp

Drawing Line BrushModeling

Do it yourself OpenGL (2x and 3x)

Modelingrendering programs Breeze designer Ayam 3DCanvas Makehuman Alice

copy DJ Foreman 2012 30

Photography (film) Silver halide crystals (AgCl AgBr AgI)

Exposure to light turns them black Developer removes the loose (exposed) halide leaving

pure silver (a negative image) Fixing bath flushes out unexposed AgHalide

Greatest desire of photographers - a medium with High speed (gathers light quickly) Low noise (no undesired artifacts) High resolution (very detailed images)

Visible lines per mm (lpm) at a specified contrast level

Notes Larger ldquoclumpsrdquo are faster but limit resolution Smaller ldquoclumpsrdquo are hard to coat evenly Randomness of coating prevents Moireacute patternshttpenwikipediaorgwikiMoirC3A9_pattern

copy DJ Foreman 2012 31

FYI ndash a little chemistry

Ag+ X- + hv rarr Ag+ + X + e-

1 Radiation frees the Br2 1 Ag ion + 1 electron (Ag+ + e- rarr Ag0) -gt 1 Ag atom3 Neighbors are now ldquosensitizedrdquo 4 Developing (Reducing) converts exposed AgX to

black metallic Ag and free X5 Remaining AgX and free X are washed away (after

mixing with thiosulfate to make it soluble)

This is one silver (Ag) ldquograinrdquo in the ldquolatent imagerdquo X is any halide (Br Cl etc)

copy DJ Foreman 2012 32

crystal

Radiation (light)

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Audio Encoding EnumerationsAUDIO_FILE_ENCODING_MULAW_8 (1) 8-bit ISDN u-law

AUDIO_FILE_ENCODING_LINEAR_8 (2) 8-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_16 (3) 16-bit linear PCM

AUDIO_FILE_ENCODING_LINEAR_32 (5) 32-bit linear PCM

AUDIO_FILE_ENCODING_FLOAT (6) 32-bit IEEE floating point

AUDIO_FILE_ENCODING_DOUBLE (7) 64-bit IEEE floating point

AUDIO_FILE_ENCODING_ADPCM_G721 (23) 4-bit CCITT g721 ADPCM

AUDIO_FILE_ENCODING_ADPCM_G723_3 (25) CCITT g723 3-bit ADPCM

AUDIO_FILE_ENCODING_ALAW_8 (27) 8-bit ISDN A-law

copy DJ Foreman 2012 20

ldquoLinearrdquo values are SIGNED intrsquosFloats are signed zero-centered normalized to ( -10 lt= x lt= 10 )

Example 2 ZIP files (PkWare)Purpose Size in bytes

Signature header 4

Required version 2

GP flags 2

Method 2

Last mod time 2

Last mod date 2

CRC-32 4

Compressed size 4

Uncompressed size 4

Filename length 2

Extra field length 2

File name variable

extra variable

copy DJ Foreman 2012 21

Ref httplivedocsadobecomflex3htmlhelphtmlcontent=ByteArrays_3html

More info on encoding httpwwwpkwarecomdocumentscase

studiesAPPNOTETXT

httpwwwfileinfocomfiletypescompressed

copy DJ Foreman 2012 22

Recording Bit-Rate

Constant (CBR) rate at which a codecs output data should be

consumed is constant Max bit-rate matters not the average Uses all available bandwidth Not good for storage (lossy)

Variable (VBR) Quantity of datatime unit (for output) varies Better quality for audio and video Slower to encode Supported by most portable devices post 2006

copy DJ Foreman 2012 23

Lab assignment ndash 1 ndash part 1

Write a program with 1 parameter If you are on an encoding team

Open a test file (eg ldquoxyztxtrdquo) Compress it using simple Huffman compression using

the frequency table from my FTP site Output compressed file (eg ldquoxyzencrdquo) to SAME folder

as input

If you are on a decoding team Open compressed file ldquoxyzencrdquo Decompress the input file Output file ldquoxyzdecrdquo so it can be compared to ldquoxyztxtrdquo

The parameter is the full input file path amp name (ie file names MUST NOT be hard-coded)

Remember a code can be a single bit

copy DJ Foreman 2012 24

25

Lab assignment 1 ndash part 2

Use ONLY MY TABLE for the frequencies Use ONLY uppercase letters of the table Do NOT use the special character

tables Find someone in class with the

opposite program and TEST your program to see if it properly encodes or decodes a message

Grading tree construction tree following proper binary output (encoders) proper character output (decoders)

copy DJ Foreman 2012

Working with ImagesModeling

Containers

A container is a FILE format NOT a code scheme

Eg AVI amp WAV are container formats Formats for storage Independent of encoding or content

Others Ogg ASF QuickTime RealMedia Matroska

DivX and MP4

copy DJ Foreman 2012 27

28

Movie (metadata)

Track

Media

Media Information

Container file format (MP4QT)

Chunk map

Comp time

Chunk offset

Key frame

Desc

Decode time

Priority map

Size map

Sample Table

Media Handler

Data information

Media header

Video header

Track header

Movie header

copy DJ Foreman 2012

Pixels A pixel is 3 color-dots (squares) in a collection of dots

(squares) making up the picture Each color-dot is for one of the three primary colors

RGB Minimum of 1 byte per color (depending on bit-depth

which varies with camera) thus at least 3 bytes per pixel

A 10MP camera (30 million color-dots) needs 30M bytes This gets compressed by the camera (JPEG format) Final file size = 30 million bytes divided by the amount

of compression (compression ratio) An 81 compression ratio would reduce that 30 million

bytes to 375 megabytes (as done on a 4 MP camera) A RAW file has NO compression

copy DJ Foreman 2012 29

Image creation

Basic mechanisms Photography

Film Digital ndashPhotoshop Gimp

Drawing Line BrushModeling

Do it yourself OpenGL (2x and 3x)

Modelingrendering programs Breeze designer Ayam 3DCanvas Makehuman Alice

copy DJ Foreman 2012 30

Photography (film) Silver halide crystals (AgCl AgBr AgI)

Exposure to light turns them black Developer removes the loose (exposed) halide leaving

pure silver (a negative image) Fixing bath flushes out unexposed AgHalide

Greatest desire of photographers - a medium with High speed (gathers light quickly) Low noise (no undesired artifacts) High resolution (very detailed images)

Visible lines per mm (lpm) at a specified contrast level

Notes Larger ldquoclumpsrdquo are faster but limit resolution Smaller ldquoclumpsrdquo are hard to coat evenly Randomness of coating prevents Moireacute patternshttpenwikipediaorgwikiMoirC3A9_pattern

copy DJ Foreman 2012 31

FYI ndash a little chemistry

Ag+ X- + hv rarr Ag+ + X + e-

1 Radiation frees the Br2 1 Ag ion + 1 electron (Ag+ + e- rarr Ag0) -gt 1 Ag atom3 Neighbors are now ldquosensitizedrdquo 4 Developing (Reducing) converts exposed AgX to

black metallic Ag and free X5 Remaining AgX and free X are washed away (after

mixing with thiosulfate to make it soluble)

This is one silver (Ag) ldquograinrdquo in the ldquolatent imagerdquo X is any halide (Br Cl etc)

copy DJ Foreman 2012 32

crystal

Radiation (light)

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Example 2 ZIP files (PkWare)Purpose Size in bytes

Signature header 4

Required version 2

GP flags 2

Method 2

Last mod time 2

Last mod date 2

CRC-32 4

Compressed size 4

Uncompressed size 4

Filename length 2

Extra field length 2

File name variable

extra variable

copy DJ Foreman 2012 21

Ref httplivedocsadobecomflex3htmlhelphtmlcontent=ByteArrays_3html

More info on encoding httpwwwpkwarecomdocumentscase

studiesAPPNOTETXT

httpwwwfileinfocomfiletypescompressed

copy DJ Foreman 2012 22

Recording Bit-Rate

Constant (CBR) rate at which a codecs output data should be

consumed is constant Max bit-rate matters not the average Uses all available bandwidth Not good for storage (lossy)

Variable (VBR) Quantity of datatime unit (for output) varies Better quality for audio and video Slower to encode Supported by most portable devices post 2006

copy DJ Foreman 2012 23

Lab assignment ndash 1 ndash part 1

Write a program with 1 parameter If you are on an encoding team

Open a test file (eg ldquoxyztxtrdquo) Compress it using simple Huffman compression using

the frequency table from my FTP site Output compressed file (eg ldquoxyzencrdquo) to SAME folder

as input

If you are on a decoding team Open compressed file ldquoxyzencrdquo Decompress the input file Output file ldquoxyzdecrdquo so it can be compared to ldquoxyztxtrdquo

The parameter is the full input file path amp name (ie file names MUST NOT be hard-coded)

Remember a code can be a single bit

copy DJ Foreman 2012 24

25

Lab assignment 1 ndash part 2

Use ONLY MY TABLE for the frequencies Use ONLY uppercase letters of the table Do NOT use the special character

tables Find someone in class with the

opposite program and TEST your program to see if it properly encodes or decodes a message

Grading tree construction tree following proper binary output (encoders) proper character output (decoders)

copy DJ Foreman 2012

Working with ImagesModeling

Containers

A container is a FILE format NOT a code scheme

Eg AVI amp WAV are container formats Formats for storage Independent of encoding or content

Others Ogg ASF QuickTime RealMedia Matroska

DivX and MP4

copy DJ Foreman 2012 27

28

Movie (metadata)

Track

Media

Media Information

Container file format (MP4QT)

Chunk map

Comp time

Chunk offset

Key frame

Desc

Decode time

Priority map

Size map

Sample Table

Media Handler

Data information

Media header

Video header

Track header

Movie header

copy DJ Foreman 2012

Pixels A pixel is 3 color-dots (squares) in a collection of dots

(squares) making up the picture Each color-dot is for one of the three primary colors

RGB Minimum of 1 byte per color (depending on bit-depth

which varies with camera) thus at least 3 bytes per pixel

A 10MP camera (30 million color-dots) needs 30M bytes This gets compressed by the camera (JPEG format) Final file size = 30 million bytes divided by the amount

of compression (compression ratio) An 81 compression ratio would reduce that 30 million

bytes to 375 megabytes (as done on a 4 MP camera) A RAW file has NO compression

copy DJ Foreman 2012 29

Image creation

Basic mechanisms Photography

Film Digital ndashPhotoshop Gimp

Drawing Line BrushModeling

Do it yourself OpenGL (2x and 3x)

Modelingrendering programs Breeze designer Ayam 3DCanvas Makehuman Alice

copy DJ Foreman 2012 30

Photography (film) Silver halide crystals (AgCl AgBr AgI)

Exposure to light turns them black Developer removes the loose (exposed) halide leaving

pure silver (a negative image) Fixing bath flushes out unexposed AgHalide

Greatest desire of photographers - a medium with High speed (gathers light quickly) Low noise (no undesired artifacts) High resolution (very detailed images)

Visible lines per mm (lpm) at a specified contrast level

Notes Larger ldquoclumpsrdquo are faster but limit resolution Smaller ldquoclumpsrdquo are hard to coat evenly Randomness of coating prevents Moireacute patternshttpenwikipediaorgwikiMoirC3A9_pattern

copy DJ Foreman 2012 31

FYI ndash a little chemistry

Ag+ X- + hv rarr Ag+ + X + e-

1 Radiation frees the Br2 1 Ag ion + 1 electron (Ag+ + e- rarr Ag0) -gt 1 Ag atom3 Neighbors are now ldquosensitizedrdquo 4 Developing (Reducing) converts exposed AgX to

black metallic Ag and free X5 Remaining AgX and free X are washed away (after

mixing with thiosulfate to make it soluble)

This is one silver (Ag) ldquograinrdquo in the ldquolatent imagerdquo X is any halide (Br Cl etc)

copy DJ Foreman 2012 32

crystal

Radiation (light)

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

More info on encoding httpwwwpkwarecomdocumentscase

studiesAPPNOTETXT

httpwwwfileinfocomfiletypescompressed

copy DJ Foreman 2012 22

Recording Bit-Rate

Constant (CBR) rate at which a codecs output data should be

consumed is constant Max bit-rate matters not the average Uses all available bandwidth Not good for storage (lossy)

Variable (VBR) Quantity of datatime unit (for output) varies Better quality for audio and video Slower to encode Supported by most portable devices post 2006

copy DJ Foreman 2012 23

Lab assignment ndash 1 ndash part 1

Write a program with 1 parameter If you are on an encoding team

Open a test file (eg ldquoxyztxtrdquo) Compress it using simple Huffman compression using

the frequency table from my FTP site Output compressed file (eg ldquoxyzencrdquo) to SAME folder

as input

If you are on a decoding team Open compressed file ldquoxyzencrdquo Decompress the input file Output file ldquoxyzdecrdquo so it can be compared to ldquoxyztxtrdquo

The parameter is the full input file path amp name (ie file names MUST NOT be hard-coded)

Remember a code can be a single bit

copy DJ Foreman 2012 24

25

Lab assignment 1 ndash part 2

Use ONLY MY TABLE for the frequencies Use ONLY uppercase letters of the table Do NOT use the special character

tables Find someone in class with the

opposite program and TEST your program to see if it properly encodes or decodes a message

Grading tree construction tree following proper binary output (encoders) proper character output (decoders)

copy DJ Foreman 2012

Working with ImagesModeling

Containers

A container is a FILE format NOT a code scheme

Eg AVI amp WAV are container formats Formats for storage Independent of encoding or content

Others Ogg ASF QuickTime RealMedia Matroska

DivX and MP4

copy DJ Foreman 2012 27

28

Movie (metadata)

Track

Media

Media Information

Container file format (MP4QT)

Chunk map

Comp time

Chunk offset

Key frame

Desc

Decode time

Priority map

Size map

Sample Table

Media Handler

Data information

Media header

Video header

Track header

Movie header

copy DJ Foreman 2012

Pixels A pixel is 3 color-dots (squares) in a collection of dots

(squares) making up the picture Each color-dot is for one of the three primary colors

RGB Minimum of 1 byte per color (depending on bit-depth

which varies with camera) thus at least 3 bytes per pixel

A 10MP camera (30 million color-dots) needs 30M bytes This gets compressed by the camera (JPEG format) Final file size = 30 million bytes divided by the amount

of compression (compression ratio) An 81 compression ratio would reduce that 30 million

bytes to 375 megabytes (as done on a 4 MP camera) A RAW file has NO compression

copy DJ Foreman 2012 29

Image creation

Basic mechanisms Photography

Film Digital ndashPhotoshop Gimp

Drawing Line BrushModeling

Do it yourself OpenGL (2x and 3x)

Modelingrendering programs Breeze designer Ayam 3DCanvas Makehuman Alice

copy DJ Foreman 2012 30

Photography (film) Silver halide crystals (AgCl AgBr AgI)

Exposure to light turns them black Developer removes the loose (exposed) halide leaving

pure silver (a negative image) Fixing bath flushes out unexposed AgHalide

Greatest desire of photographers - a medium with High speed (gathers light quickly) Low noise (no undesired artifacts) High resolution (very detailed images)

Visible lines per mm (lpm) at a specified contrast level

Notes Larger ldquoclumpsrdquo are faster but limit resolution Smaller ldquoclumpsrdquo are hard to coat evenly Randomness of coating prevents Moireacute patternshttpenwikipediaorgwikiMoirC3A9_pattern

copy DJ Foreman 2012 31

FYI ndash a little chemistry

Ag+ X- + hv rarr Ag+ + X + e-

1 Radiation frees the Br2 1 Ag ion + 1 electron (Ag+ + e- rarr Ag0) -gt 1 Ag atom3 Neighbors are now ldquosensitizedrdquo 4 Developing (Reducing) converts exposed AgX to

black metallic Ag and free X5 Remaining AgX and free X are washed away (after

mixing with thiosulfate to make it soluble)

This is one silver (Ag) ldquograinrdquo in the ldquolatent imagerdquo X is any halide (Br Cl etc)

copy DJ Foreman 2012 32

crystal

Radiation (light)

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Recording Bit-Rate

Constant (CBR) rate at which a codecs output data should be

consumed is constant Max bit-rate matters not the average Uses all available bandwidth Not good for storage (lossy)

Variable (VBR) Quantity of datatime unit (for output) varies Better quality for audio and video Slower to encode Supported by most portable devices post 2006

copy DJ Foreman 2012 23

Lab assignment ndash 1 ndash part 1

Write a program with 1 parameter If you are on an encoding team

Open a test file (eg ldquoxyztxtrdquo) Compress it using simple Huffman compression using

the frequency table from my FTP site Output compressed file (eg ldquoxyzencrdquo) to SAME folder

as input

If you are on a decoding team Open compressed file ldquoxyzencrdquo Decompress the input file Output file ldquoxyzdecrdquo so it can be compared to ldquoxyztxtrdquo

The parameter is the full input file path amp name (ie file names MUST NOT be hard-coded)

Remember a code can be a single bit

copy DJ Foreman 2012 24

25

Lab assignment 1 ndash part 2

Use ONLY MY TABLE for the frequencies Use ONLY uppercase letters of the table Do NOT use the special character

tables Find someone in class with the

opposite program and TEST your program to see if it properly encodes or decodes a message

Grading tree construction tree following proper binary output (encoders) proper character output (decoders)

copy DJ Foreman 2012

Working with ImagesModeling

Containers

A container is a FILE format NOT a code scheme

Eg AVI amp WAV are container formats Formats for storage Independent of encoding or content

Others Ogg ASF QuickTime RealMedia Matroska

DivX and MP4

copy DJ Foreman 2012 27

28

Movie (metadata)

Track

Media

Media Information

Container file format (MP4QT)

Chunk map

Comp time

Chunk offset

Key frame

Desc

Decode time

Priority map

Size map

Sample Table

Media Handler

Data information

Media header

Video header

Track header

Movie header

copy DJ Foreman 2012

Pixels A pixel is 3 color-dots (squares) in a collection of dots

(squares) making up the picture Each color-dot is for one of the three primary colors

RGB Minimum of 1 byte per color (depending on bit-depth

which varies with camera) thus at least 3 bytes per pixel

A 10MP camera (30 million color-dots) needs 30M bytes This gets compressed by the camera (JPEG format) Final file size = 30 million bytes divided by the amount

of compression (compression ratio) An 81 compression ratio would reduce that 30 million

bytes to 375 megabytes (as done on a 4 MP camera) A RAW file has NO compression

copy DJ Foreman 2012 29

Image creation

Basic mechanisms Photography

Film Digital ndashPhotoshop Gimp

Drawing Line BrushModeling

Do it yourself OpenGL (2x and 3x)

Modelingrendering programs Breeze designer Ayam 3DCanvas Makehuman Alice

copy DJ Foreman 2012 30

Photography (film) Silver halide crystals (AgCl AgBr AgI)

Exposure to light turns them black Developer removes the loose (exposed) halide leaving

pure silver (a negative image) Fixing bath flushes out unexposed AgHalide

Greatest desire of photographers - a medium with High speed (gathers light quickly) Low noise (no undesired artifacts) High resolution (very detailed images)

Visible lines per mm (lpm) at a specified contrast level

Notes Larger ldquoclumpsrdquo are faster but limit resolution Smaller ldquoclumpsrdquo are hard to coat evenly Randomness of coating prevents Moireacute patternshttpenwikipediaorgwikiMoirC3A9_pattern

copy DJ Foreman 2012 31

FYI ndash a little chemistry

Ag+ X- + hv rarr Ag+ + X + e-

1 Radiation frees the Br2 1 Ag ion + 1 electron (Ag+ + e- rarr Ag0) -gt 1 Ag atom3 Neighbors are now ldquosensitizedrdquo 4 Developing (Reducing) converts exposed AgX to

black metallic Ag and free X5 Remaining AgX and free X are washed away (after

mixing with thiosulfate to make it soluble)

This is one silver (Ag) ldquograinrdquo in the ldquolatent imagerdquo X is any halide (Br Cl etc)

copy DJ Foreman 2012 32

crystal

Radiation (light)

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Lab assignment ndash 1 ndash part 1

Write a program with 1 parameter If you are on an encoding team

Open a test file (eg ldquoxyztxtrdquo) Compress it using simple Huffman compression using

the frequency table from my FTP site Output compressed file (eg ldquoxyzencrdquo) to SAME folder

as input

If you are on a decoding team Open compressed file ldquoxyzencrdquo Decompress the input file Output file ldquoxyzdecrdquo so it can be compared to ldquoxyztxtrdquo

The parameter is the full input file path amp name (ie file names MUST NOT be hard-coded)

Remember a code can be a single bit

copy DJ Foreman 2012 24

25

Lab assignment 1 ndash part 2

Use ONLY MY TABLE for the frequencies Use ONLY uppercase letters of the table Do NOT use the special character

tables Find someone in class with the

opposite program and TEST your program to see if it properly encodes or decodes a message

Grading tree construction tree following proper binary output (encoders) proper character output (decoders)

copy DJ Foreman 2012

Working with ImagesModeling

Containers

A container is a FILE format NOT a code scheme

Eg AVI amp WAV are container formats Formats for storage Independent of encoding or content

Others Ogg ASF QuickTime RealMedia Matroska

DivX and MP4

copy DJ Foreman 2012 27

28

Movie (metadata)

Track

Media

Media Information

Container file format (MP4QT)

Chunk map

Comp time

Chunk offset

Key frame

Desc

Decode time

Priority map

Size map

Sample Table

Media Handler

Data information

Media header

Video header

Track header

Movie header

copy DJ Foreman 2012

Pixels A pixel is 3 color-dots (squares) in a collection of dots

(squares) making up the picture Each color-dot is for one of the three primary colors

RGB Minimum of 1 byte per color (depending on bit-depth

which varies with camera) thus at least 3 bytes per pixel

A 10MP camera (30 million color-dots) needs 30M bytes This gets compressed by the camera (JPEG format) Final file size = 30 million bytes divided by the amount

of compression (compression ratio) An 81 compression ratio would reduce that 30 million

bytes to 375 megabytes (as done on a 4 MP camera) A RAW file has NO compression

copy DJ Foreman 2012 29

Image creation

Basic mechanisms Photography

Film Digital ndashPhotoshop Gimp

Drawing Line BrushModeling

Do it yourself OpenGL (2x and 3x)

Modelingrendering programs Breeze designer Ayam 3DCanvas Makehuman Alice

copy DJ Foreman 2012 30

Photography (film) Silver halide crystals (AgCl AgBr AgI)

Exposure to light turns them black Developer removes the loose (exposed) halide leaving

pure silver (a negative image) Fixing bath flushes out unexposed AgHalide

Greatest desire of photographers - a medium with High speed (gathers light quickly) Low noise (no undesired artifacts) High resolution (very detailed images)

Visible lines per mm (lpm) at a specified contrast level

Notes Larger ldquoclumpsrdquo are faster but limit resolution Smaller ldquoclumpsrdquo are hard to coat evenly Randomness of coating prevents Moireacute patternshttpenwikipediaorgwikiMoirC3A9_pattern

copy DJ Foreman 2012 31

FYI ndash a little chemistry

Ag+ X- + hv rarr Ag+ + X + e-

1 Radiation frees the Br2 1 Ag ion + 1 electron (Ag+ + e- rarr Ag0) -gt 1 Ag atom3 Neighbors are now ldquosensitizedrdquo 4 Developing (Reducing) converts exposed AgX to

black metallic Ag and free X5 Remaining AgX and free X are washed away (after

mixing with thiosulfate to make it soluble)

This is one silver (Ag) ldquograinrdquo in the ldquolatent imagerdquo X is any halide (Br Cl etc)

copy DJ Foreman 2012 32

crystal

Radiation (light)

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

25

Lab assignment 1 ndash part 2

Use ONLY MY TABLE for the frequencies Use ONLY uppercase letters of the table Do NOT use the special character

tables Find someone in class with the

opposite program and TEST your program to see if it properly encodes or decodes a message

Grading tree construction tree following proper binary output (encoders) proper character output (decoders)

copy DJ Foreman 2012

Working with ImagesModeling

Containers

A container is a FILE format NOT a code scheme

Eg AVI amp WAV are container formats Formats for storage Independent of encoding or content

Others Ogg ASF QuickTime RealMedia Matroska

DivX and MP4

copy DJ Foreman 2012 27

28

Movie (metadata)

Track

Media

Media Information

Container file format (MP4QT)

Chunk map

Comp time

Chunk offset

Key frame

Desc

Decode time

Priority map

Size map

Sample Table

Media Handler

Data information

Media header

Video header

Track header

Movie header

copy DJ Foreman 2012

Pixels A pixel is 3 color-dots (squares) in a collection of dots

(squares) making up the picture Each color-dot is for one of the three primary colors

RGB Minimum of 1 byte per color (depending on bit-depth

which varies with camera) thus at least 3 bytes per pixel

A 10MP camera (30 million color-dots) needs 30M bytes This gets compressed by the camera (JPEG format) Final file size = 30 million bytes divided by the amount

of compression (compression ratio) An 81 compression ratio would reduce that 30 million

bytes to 375 megabytes (as done on a 4 MP camera) A RAW file has NO compression

copy DJ Foreman 2012 29

Image creation

Basic mechanisms Photography

Film Digital ndashPhotoshop Gimp

Drawing Line BrushModeling

Do it yourself OpenGL (2x and 3x)

Modelingrendering programs Breeze designer Ayam 3DCanvas Makehuman Alice

copy DJ Foreman 2012 30

Photography (film) Silver halide crystals (AgCl AgBr AgI)

Exposure to light turns them black Developer removes the loose (exposed) halide leaving

pure silver (a negative image) Fixing bath flushes out unexposed AgHalide

Greatest desire of photographers - a medium with High speed (gathers light quickly) Low noise (no undesired artifacts) High resolution (very detailed images)

Visible lines per mm (lpm) at a specified contrast level

Notes Larger ldquoclumpsrdquo are faster but limit resolution Smaller ldquoclumpsrdquo are hard to coat evenly Randomness of coating prevents Moireacute patternshttpenwikipediaorgwikiMoirC3A9_pattern

copy DJ Foreman 2012 31

FYI ndash a little chemistry

Ag+ X- + hv rarr Ag+ + X + e-

1 Radiation frees the Br2 1 Ag ion + 1 electron (Ag+ + e- rarr Ag0) -gt 1 Ag atom3 Neighbors are now ldquosensitizedrdquo 4 Developing (Reducing) converts exposed AgX to

black metallic Ag and free X5 Remaining AgX and free X are washed away (after

mixing with thiosulfate to make it soluble)

This is one silver (Ag) ldquograinrdquo in the ldquolatent imagerdquo X is any halide (Br Cl etc)

copy DJ Foreman 2012 32

crystal

Radiation (light)

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Working with ImagesModeling

Containers

A container is a FILE format NOT a code scheme

Eg AVI amp WAV are container formats Formats for storage Independent of encoding or content

Others Ogg ASF QuickTime RealMedia Matroska

DivX and MP4

copy DJ Foreman 2012 27

28

Movie (metadata)

Track

Media

Media Information

Container file format (MP4QT)

Chunk map

Comp time

Chunk offset

Key frame

Desc

Decode time

Priority map

Size map

Sample Table

Media Handler

Data information

Media header

Video header

Track header

Movie header

copy DJ Foreman 2012

Pixels A pixel is 3 color-dots (squares) in a collection of dots

(squares) making up the picture Each color-dot is for one of the three primary colors

RGB Minimum of 1 byte per color (depending on bit-depth

which varies with camera) thus at least 3 bytes per pixel

A 10MP camera (30 million color-dots) needs 30M bytes This gets compressed by the camera (JPEG format) Final file size = 30 million bytes divided by the amount

of compression (compression ratio) An 81 compression ratio would reduce that 30 million

bytes to 375 megabytes (as done on a 4 MP camera) A RAW file has NO compression

copy DJ Foreman 2012 29

Image creation

Basic mechanisms Photography

Film Digital ndashPhotoshop Gimp

Drawing Line BrushModeling

Do it yourself OpenGL (2x and 3x)

Modelingrendering programs Breeze designer Ayam 3DCanvas Makehuman Alice

copy DJ Foreman 2012 30

Photography (film) Silver halide crystals (AgCl AgBr AgI)

Exposure to light turns them black Developer removes the loose (exposed) halide leaving

pure silver (a negative image) Fixing bath flushes out unexposed AgHalide

Greatest desire of photographers - a medium with High speed (gathers light quickly) Low noise (no undesired artifacts) High resolution (very detailed images)

Visible lines per mm (lpm) at a specified contrast level

Notes Larger ldquoclumpsrdquo are faster but limit resolution Smaller ldquoclumpsrdquo are hard to coat evenly Randomness of coating prevents Moireacute patternshttpenwikipediaorgwikiMoirC3A9_pattern

copy DJ Foreman 2012 31

FYI ndash a little chemistry

Ag+ X- + hv rarr Ag+ + X + e-

1 Radiation frees the Br2 1 Ag ion + 1 electron (Ag+ + e- rarr Ag0) -gt 1 Ag atom3 Neighbors are now ldquosensitizedrdquo 4 Developing (Reducing) converts exposed AgX to

black metallic Ag and free X5 Remaining AgX and free X are washed away (after

mixing with thiosulfate to make it soluble)

This is one silver (Ag) ldquograinrdquo in the ldquolatent imagerdquo X is any halide (Br Cl etc)

copy DJ Foreman 2012 32

crystal

Radiation (light)

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Containers

A container is a FILE format NOT a code scheme

Eg AVI amp WAV are container formats Formats for storage Independent of encoding or content

Others Ogg ASF QuickTime RealMedia Matroska

DivX and MP4

copy DJ Foreman 2012 27

28

Movie (metadata)

Track

Media

Media Information

Container file format (MP4QT)

Chunk map

Comp time

Chunk offset

Key frame

Desc

Decode time

Priority map

Size map

Sample Table

Media Handler

Data information

Media header

Video header

Track header

Movie header

copy DJ Foreman 2012

Pixels A pixel is 3 color-dots (squares) in a collection of dots

(squares) making up the picture Each color-dot is for one of the three primary colors

RGB Minimum of 1 byte per color (depending on bit-depth

which varies with camera) thus at least 3 bytes per pixel

A 10MP camera (30 million color-dots) needs 30M bytes This gets compressed by the camera (JPEG format) Final file size = 30 million bytes divided by the amount

of compression (compression ratio) An 81 compression ratio would reduce that 30 million

bytes to 375 megabytes (as done on a 4 MP camera) A RAW file has NO compression

copy DJ Foreman 2012 29

Image creation

Basic mechanisms Photography

Film Digital ndashPhotoshop Gimp

Drawing Line BrushModeling

Do it yourself OpenGL (2x and 3x)

Modelingrendering programs Breeze designer Ayam 3DCanvas Makehuman Alice

copy DJ Foreman 2012 30

Photography (film) Silver halide crystals (AgCl AgBr AgI)

Exposure to light turns them black Developer removes the loose (exposed) halide leaving

pure silver (a negative image) Fixing bath flushes out unexposed AgHalide

Greatest desire of photographers - a medium with High speed (gathers light quickly) Low noise (no undesired artifacts) High resolution (very detailed images)

Visible lines per mm (lpm) at a specified contrast level

Notes Larger ldquoclumpsrdquo are faster but limit resolution Smaller ldquoclumpsrdquo are hard to coat evenly Randomness of coating prevents Moireacute patternshttpenwikipediaorgwikiMoirC3A9_pattern

copy DJ Foreman 2012 31

FYI ndash a little chemistry

Ag+ X- + hv rarr Ag+ + X + e-

1 Radiation frees the Br2 1 Ag ion + 1 electron (Ag+ + e- rarr Ag0) -gt 1 Ag atom3 Neighbors are now ldquosensitizedrdquo 4 Developing (Reducing) converts exposed AgX to

black metallic Ag and free X5 Remaining AgX and free X are washed away (after

mixing with thiosulfate to make it soluble)

This is one silver (Ag) ldquograinrdquo in the ldquolatent imagerdquo X is any halide (Br Cl etc)

copy DJ Foreman 2012 32

crystal

Radiation (light)

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

28

Movie (metadata)

Track

Media

Media Information

Container file format (MP4QT)

Chunk map

Comp time

Chunk offset

Key frame

Desc

Decode time

Priority map

Size map

Sample Table

Media Handler

Data information

Media header

Video header

Track header

Movie header

copy DJ Foreman 2012

Pixels A pixel is 3 color-dots (squares) in a collection of dots

(squares) making up the picture Each color-dot is for one of the three primary colors

RGB Minimum of 1 byte per color (depending on bit-depth

which varies with camera) thus at least 3 bytes per pixel

A 10MP camera (30 million color-dots) needs 30M bytes This gets compressed by the camera (JPEG format) Final file size = 30 million bytes divided by the amount

of compression (compression ratio) An 81 compression ratio would reduce that 30 million

bytes to 375 megabytes (as done on a 4 MP camera) A RAW file has NO compression

copy DJ Foreman 2012 29

Image creation

Basic mechanisms Photography

Film Digital ndashPhotoshop Gimp

Drawing Line BrushModeling

Do it yourself OpenGL (2x and 3x)

Modelingrendering programs Breeze designer Ayam 3DCanvas Makehuman Alice

copy DJ Foreman 2012 30

Photography (film) Silver halide crystals (AgCl AgBr AgI)

Exposure to light turns them black Developer removes the loose (exposed) halide leaving

pure silver (a negative image) Fixing bath flushes out unexposed AgHalide

Greatest desire of photographers - a medium with High speed (gathers light quickly) Low noise (no undesired artifacts) High resolution (very detailed images)

Visible lines per mm (lpm) at a specified contrast level

Notes Larger ldquoclumpsrdquo are faster but limit resolution Smaller ldquoclumpsrdquo are hard to coat evenly Randomness of coating prevents Moireacute patternshttpenwikipediaorgwikiMoirC3A9_pattern

copy DJ Foreman 2012 31

FYI ndash a little chemistry

Ag+ X- + hv rarr Ag+ + X + e-

1 Radiation frees the Br2 1 Ag ion + 1 electron (Ag+ + e- rarr Ag0) -gt 1 Ag atom3 Neighbors are now ldquosensitizedrdquo 4 Developing (Reducing) converts exposed AgX to

black metallic Ag and free X5 Remaining AgX and free X are washed away (after

mixing with thiosulfate to make it soluble)

This is one silver (Ag) ldquograinrdquo in the ldquolatent imagerdquo X is any halide (Br Cl etc)

copy DJ Foreman 2012 32

crystal

Radiation (light)

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Pixels A pixel is 3 color-dots (squares) in a collection of dots

(squares) making up the picture Each color-dot is for one of the three primary colors

RGB Minimum of 1 byte per color (depending on bit-depth

which varies with camera) thus at least 3 bytes per pixel

A 10MP camera (30 million color-dots) needs 30M bytes This gets compressed by the camera (JPEG format) Final file size = 30 million bytes divided by the amount

of compression (compression ratio) An 81 compression ratio would reduce that 30 million

bytes to 375 megabytes (as done on a 4 MP camera) A RAW file has NO compression

copy DJ Foreman 2012 29

Image creation

Basic mechanisms Photography

Film Digital ndashPhotoshop Gimp

Drawing Line BrushModeling

Do it yourself OpenGL (2x and 3x)

Modelingrendering programs Breeze designer Ayam 3DCanvas Makehuman Alice

copy DJ Foreman 2012 30

Photography (film) Silver halide crystals (AgCl AgBr AgI)

Exposure to light turns them black Developer removes the loose (exposed) halide leaving

pure silver (a negative image) Fixing bath flushes out unexposed AgHalide

Greatest desire of photographers - a medium with High speed (gathers light quickly) Low noise (no undesired artifacts) High resolution (very detailed images)

Visible lines per mm (lpm) at a specified contrast level

Notes Larger ldquoclumpsrdquo are faster but limit resolution Smaller ldquoclumpsrdquo are hard to coat evenly Randomness of coating prevents Moireacute patternshttpenwikipediaorgwikiMoirC3A9_pattern

copy DJ Foreman 2012 31

FYI ndash a little chemistry

Ag+ X- + hv rarr Ag+ + X + e-

1 Radiation frees the Br2 1 Ag ion + 1 electron (Ag+ + e- rarr Ag0) -gt 1 Ag atom3 Neighbors are now ldquosensitizedrdquo 4 Developing (Reducing) converts exposed AgX to

black metallic Ag and free X5 Remaining AgX and free X are washed away (after

mixing with thiosulfate to make it soluble)

This is one silver (Ag) ldquograinrdquo in the ldquolatent imagerdquo X is any halide (Br Cl etc)

copy DJ Foreman 2012 32

crystal

Radiation (light)

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Image creation

Basic mechanisms Photography

Film Digital ndashPhotoshop Gimp

Drawing Line BrushModeling

Do it yourself OpenGL (2x and 3x)

Modelingrendering programs Breeze designer Ayam 3DCanvas Makehuman Alice

copy DJ Foreman 2012 30

Photography (film) Silver halide crystals (AgCl AgBr AgI)

Exposure to light turns them black Developer removes the loose (exposed) halide leaving

pure silver (a negative image) Fixing bath flushes out unexposed AgHalide

Greatest desire of photographers - a medium with High speed (gathers light quickly) Low noise (no undesired artifacts) High resolution (very detailed images)

Visible lines per mm (lpm) at a specified contrast level

Notes Larger ldquoclumpsrdquo are faster but limit resolution Smaller ldquoclumpsrdquo are hard to coat evenly Randomness of coating prevents Moireacute patternshttpenwikipediaorgwikiMoirC3A9_pattern

copy DJ Foreman 2012 31

FYI ndash a little chemistry

Ag+ X- + hv rarr Ag+ + X + e-

1 Radiation frees the Br2 1 Ag ion + 1 electron (Ag+ + e- rarr Ag0) -gt 1 Ag atom3 Neighbors are now ldquosensitizedrdquo 4 Developing (Reducing) converts exposed AgX to

black metallic Ag and free X5 Remaining AgX and free X are washed away (after

mixing with thiosulfate to make it soluble)

This is one silver (Ag) ldquograinrdquo in the ldquolatent imagerdquo X is any halide (Br Cl etc)

copy DJ Foreman 2012 32

crystal

Radiation (light)

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Photography (film) Silver halide crystals (AgCl AgBr AgI)

Exposure to light turns them black Developer removes the loose (exposed) halide leaving

pure silver (a negative image) Fixing bath flushes out unexposed AgHalide

Greatest desire of photographers - a medium with High speed (gathers light quickly) Low noise (no undesired artifacts) High resolution (very detailed images)

Visible lines per mm (lpm) at a specified contrast level

Notes Larger ldquoclumpsrdquo are faster but limit resolution Smaller ldquoclumpsrdquo are hard to coat evenly Randomness of coating prevents Moireacute patternshttpenwikipediaorgwikiMoirC3A9_pattern

copy DJ Foreman 2012 31

FYI ndash a little chemistry

Ag+ X- + hv rarr Ag+ + X + e-

1 Radiation frees the Br2 1 Ag ion + 1 electron (Ag+ + e- rarr Ag0) -gt 1 Ag atom3 Neighbors are now ldquosensitizedrdquo 4 Developing (Reducing) converts exposed AgX to

black metallic Ag and free X5 Remaining AgX and free X are washed away (after

mixing with thiosulfate to make it soluble)

This is one silver (Ag) ldquograinrdquo in the ldquolatent imagerdquo X is any halide (Br Cl etc)

copy DJ Foreman 2012 32

crystal

Radiation (light)

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

FYI ndash a little chemistry

Ag+ X- + hv rarr Ag+ + X + e-

1 Radiation frees the Br2 1 Ag ion + 1 electron (Ag+ + e- rarr Ag0) -gt 1 Ag atom3 Neighbors are now ldquosensitizedrdquo 4 Developing (Reducing) converts exposed AgX to

black metallic Ag and free X5 Remaining AgX and free X are washed away (after

mixing with thiosulfate to make it soluble)

This is one silver (Ag) ldquograinrdquo in the ldquolatent imagerdquo X is any halide (Br Cl etc)

copy DJ Foreman 2012 32

crystal

Radiation (light)

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Resolution 35 mm film for comparison

A 36 x 24 mm frame of ISO 100-speed film contains (asymp) the equivalent of 20 million pixels 1 There are NO pixels in film

Full-frame digital (36 x 24mm = 32 image ratio) Canon EOS 5D Nikon D3 (21MP) - $5K2

Medium format digital (6 x 45cm) Phase One P40+ (39MP) ($22K)

APS-C sized sensor (asymp24x16mm = 32 image ratio) (note 2009 data below)

Nikon D90 (123MP) - $900 Canon Rebel (15MP) $700 Used in most DSLRrsquos (note image ratio 32)

Other SONY Alpha (note image ratio 43)

1 Langford Michael Basic Photography (7th Ed 2000) Oxford Focal Press ISBN 0 240 51592 7

2 Prices as of 2009 given for comparison purposes onlycopy DJ Foreman 2012 33

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Photography (digital) Image capture formats

RAW ndash 3 separate images (RGB) [12 bitspixel (4288 2848)]8=18318336 bytes per

picture JPEG ndash compressed (16x 8x 4x) Some cameras do BOTH (RAW + JPEG) for each image

Pixel notes Sensor size 35mm (or larger) vs APS-C Pixel count ndash any increase rarr

decrease in pixel size or increase in sensor size Pixel size ndash smaller pixels give us

Greater resolution (finer details) - Good More ldquonoiserdquo (incorrect values) - bad Possible ldquoadjacencyrdquo issues - bad Loss of sensitivity - bad

copy DJ Foreman 2012 34

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

35

Some Comparisons

Digital media vs film Pixels are the same (RGB) Pixel size varies with device (mfgr

dependency) Pixel density varies Zooming digitally does NOT give you more

detail (in spite of what they do on TV shows)

Optical zoom is useful Uses optical magnification of the image on

the target Expansion of light rays not multiplication of

pixelscopy DJ Foreman 2012

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Working Definitions - 1

Graphic Model data structure nodes represent random variables Pairs of nodes connected by arcs correspond

to variables that are not independent Graphic modeling

using a Graphic Model to simulate a real-world object (this is not a formal definition)

Rendering generating a 2D image from a 3D graphic model

copy DJ Foreman 2012 36

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

37

Working Definitions - 2

Geometric primitives ndash Points line segments and polygons Described by vertices

Control points ndash Special points attached to a node on a

Beacutezier curve Alter the shape and angle of adjacent curve

segment Evaluators ndash

Functions that interpolate a set of control points

Produce a new set of control pointscopy DJ Foreman 2012

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

38

Renderman Interface Bytestream (RIB) copy Pixar

A standardized interface httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

A plaintext file created by modeling programs

Input to rendering programs

copy DJ Foreman 2012

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Context

Context is the ldquoversionrdquo of OpenGL in use OpenGL 2x (current to 2009) OpenGL 30 (20 + deprecated functions) OpenGL 31 only newer graphics cards

Many old (lt= 30) functions are deprecated Many actions now done via shaders 1st create a context (like the chicken amp the egg)

create an old (eg 30) context activate it create the new context deactivate old context

Old context needed to create new one

copy DJ Foreman 2012 39

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

40

OpenGL 20 Fixed Function Pipeline

copy DJ Foreman 2012

Vertex Data

Frame Buffer

Vertex Processor

Fragment Processor

Pixel Data

Per-vertex operationsbull Eye-space

coordinatesbull Colorsbull Texture

coordinatesbull Fog coordinatesbull Point size

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

41

The Vertex Processor Lights object vertices V

Determine color to assign Pass color to pixel processor

This is then interpolated across the polygon

Transforms object vertices V Construct projection matrix

transform V from world space to camera space Transform V from camera space to screen space

Multiply vectors with matrices ie vectora column-vectorb

[x0 y0z0] [x1y1z1] x0x1 + y0y1 + z0z1

Has no information regarding connectivity Therefore canrsquot do back-face clipping

copy DJ Foreman 2012

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

42

OpenGL 20 Vertex Processor Pipeline

copy DJ Foreman 2012

Vertex Coordinates

Normal Vector

Color Values

Texture Coordinates

Fog Coordinates

Model View Matrix

Model View Matrix

Texture Matrix

Projection Matrix

Primitive Setup

Clipping

Vertex Shadersreplace everything in the dotted box

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

43

DirectX vs OpenGL

Direct3D designed as a hw

interface Application

manages hw Mostly for games

Opengl ldquomayrdquo be hardware

accelerated Implementation

manages hw Mostly for pro use

(more general-purpose)

copy DJ Foreman 2012

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

44

The Model-view Matrix

Used to transform vertices that enter the 3D API before they are rendered to the screen

Called the ldquoworld and viewrdquo matrix in DirectX

copy DJ Foreman 2012

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

45

Graphic fragment

The data needed to shade a pixel The data needed to test whether the

fragment survives Based on front-to-back relations

copy DJ Foreman 2012

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

46

Fragment Processing

copy DJ Foreman 2012

Color Summation

Per-pixel Fogging

Fragment Tests(survival etc)Frame Buffer

Texture Mapping

BitmapsPixelRectangles

From primitive setup

fragment shader replaces these

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Lab 2- basic modeling Write a program using the OpenGL 20 interface

Create 3 figures Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided 3D figure) (a pyramid)

See httpenwikipediaorgwikiPentahedron for examples ANY one Vertex pinned at 000 Allowed to be non-equilateral

Draw all 3 axes (optional but HIGHLY recommended) The triangle is within the bounds of the rectangle

(or vice-versa) both objects in same plane different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid

Cursor keys larrrarruarrdarr change camera location for world view L amp R keys rotate pyramid (CWCCW) about Z-axis Pinned pyramid Vertex STAYS at 000 The pyramid rotates the ldquocamerardquo position remains fixed

copy DJ Foreman 2012 47

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

48

Lab 2 grading

Point values Rectangle and triangle both exist

5 Pyramid exists 10 All have colors applied 5 View rotation (updown amp leftright) 10 pts

EACH Pyramid can be rotated (+ and -)

20 pts

copy DJ Foreman 2012

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

49

Some Examples for lab 2

copy DJ Foreman 2012

000

001

000

001

6-vertex pentahedronValid for lab

Still has 5 sides

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

OpenGL

PC Requirements OPENGL library - for manipulating the

model opengl32lib opengl32dll (comes with

WindowsXP amp 7) glu32dll ldquo gluh and glh ldquo

FreeGlut (newer ndash Open Source)

copy DJ Foreman 2012 50

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

51

Window Control

ONE of the following Glut - note the ldquotrdquo after the ldquoglurdquo

httpwwwopenglorgresourceslibrariesglutglut_downloadsphpwindows glut32dll rdquo glut32lib rdquo gluth

FreeGlut (newer ndash Open Source)httpfreeglutsourceforgenet freeglutdll freeglutlib gluth

gluth does a include of glh and gluhcopy DJ Foreman 2012

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Programming in OpenGL

Define window Define objects Initialize ldquocallbackrdquo functions Initialize window Run the gl main loop

Useful tutorialhttpwwwthepcspycomreadopengl__introduction_and_getting_started

Note double underscore in name abovecopy DJ Foreman 2012 52

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

OpenGL Program Structure

Two parts1 Modeling coloring etc

Use standard C code with calls to OpenGL functions

Your program runs as subroutines WITHIN the OpenGL MainLoop

2 Callback functions Called by OpenGL from OUTSIDE your program Display Reshape

Purpose of ldquoregisteringrdquo callbacks passes a pointer to your functions allows OpenGL to call them

copy DJ Foreman 2012 53

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Views Orthographic

All views are 2-dimensional Not good for games ignores the z-axis

Perspective See model from ldquocamera positionrdquo via

ldquoviewportrdquo

copy DJ Foreman 2012 54

cameraviewport

Φ

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

55

Image Rotation

Left-handed Right-handedtX2 + c tXY - sZ tXZ +

sY0 tX2 + c tXY + sZ tXZ - sY 0

tXY+sZ tY2 + c tYZ - sX 0 tXY-sZ tY2 + c tYZ + sX

0

tXZ - sY

tYZ + sX

tZ2 + c 0 tXY + sY

tYZ - sX tZ2 + c 0

0 0 0 1 0 0 0 1

copy DJ Foreman 2012

c = Cos (theta) s = Sin (theta)t = 1-Cos (theta) ltXYZgt is the unit vector for your arbitrary axis

By using the axisangle representation for your rotations it is possible to avoid gimbal lock

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Program structure

Movement functions Shape defining functions (lines

polygons etc) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)

NOTE gl glu amp glut prefixes on functionsBe careful

copy DJ Foreman 2012 56

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

57

GlutReshapeFunc

Transforms geometry for new window size

Used after every frame is drawn or after a re-size

void reshape (int w int h)

glViewport (0 0 w h)glMatrixMode (GL_PROJECTION)glLoadIdentity ( )glOrtho(hellip)glMatrixMode (GL_MODELVIEW)

copy DJ Foreman 2012

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

58

GL_Load_Identity

Replace current matrix with Identity matrix

Prevents applying multiple perspectives

copy DJ Foreman 2012

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

59

gluLookAt

Creates a viewing matrix set the camera position and orientation derived from

an eye point ndash where the viewer is located a reference point for the center of the scene an UP vector ndash maps the given vector TO the

y axis Describes any ldquotiltrdquo of the camera Defines the new value for ldquouprdquo as seen by the

viewer

copy DJ Foreman 2012

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

60

Matrix Modes

GL_PROJECTION sets up our camera GL_MODELVIEW for drawing GL_TEXTURE adjusts how textures are

drawn onto shapes

copy DJ Foreman 2012

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

61

Keyboard-related functions

glutSpecialFunc glutKeyboardFunc

copy DJ Foreman 2012

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Basic Functions in ldquomainrdquo

Register your ldquocallbackrdquo functionsglutSpecialFunc (your function name)glutKeyboardFunc (your function name)display reshape

Run the programdj_init() your init routine (if any)glutMainLoop()

copy DJ Foreman 2012 62

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

63

Skeleton of program ndash slide 1include ltGLgluthgtinclude ltstdiohgtinclude ltmathhgtinclude ltstdlibhgtfloat exeyezthetavoid leftMotion (int x int y)void rightMotion (int x int y) GLfloat pvertices [][3]= and then their colors GLfloat pcolors[][3]=

now define the polygon that USES those vertices define any ONE face of the pyramid at a time

void pface(int a int b int c) pface is a name I made up for ldquopyramid facerdquo

3 vertices define a faceglBegin(GL_POLYGON)glEnd()

copy DJ Foreman 2012

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

64

Skeleton of program ndash slide 2void draw_pyramid() implement the faces of the figure

glPolygonMode (GL_FRONT_AND_BACK GL_FILL)

void draw_square()void draw_triangle() void draw_axes()

glPushMatrix ()glBegin(GL_LINES) as pairs of points (default action of GL_LINES)glEnd()glPopMatrix ()

copy DJ Foreman 2012

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

65

Skeleton of program ndash slide 3void dj_display() callback for glutDisplayFunc this is the function that draws the graphic object in the

pre-created window glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) clear window

glLoadIdentity() eye at upgluLookAt(ex ey ez 000 010) draw_square()draw_triangle()draw_axes() last in code means display on topdraw_pyramid() now draw the pyramidglFlush()glutSwapBuffers ()

copy DJ Foreman 2012

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

66

Initialize the program ndash slide 4void dj_init() define the palette amp background

glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE)

glClearColor(00 00 00 00) set clear color to black

now specify that parameters will be RGB values amp opacity

glColor4f(00 04 05 05)

copy DJ Foreman 2012

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

67

Create a colored line ndash slide 5Void draw_a_line()

glPushMatrix ()glBegin(GL_LINES)

glColor3f(075 00 00) RGBglVertex3f(00 00 00) point 1 (xyz

coordinates) glVertex3f(150 00 00) point 2

glEnd()glPopMatrix ()

NOTE function names end in ldquo3frdquo

3=number of argumentsf=floating point values

copy DJ Foreman 2012

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

68

Skeleton of program ndash slide 6int main(int argc charargv[])

glutInit(ampargc argv) donrsquot leave this out

glutInitWindowSize(wdht)

glutInitWindowPosition(wxwy)

glutCreateWindow(DJs 1st) define name on window

glutSpecialFunc(myspecialkeys)

register the call-back functions with Glut

glutKeyboardFunc(mykey)

glutDisplayFunc(dj_display) more setup

glutReshapeFunc(dj_reshaper)

dj_init()

Some texts show init being called AFTER the glutDisplayFunc call

but it doesnt actually CAUSE display action it just sets up the environment

glutMainLoop() the only real executable stmts in program

return 0 (except if for etc)

copy DJ Foreman 2012

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Basic Display Function

glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)

glLoadIdentity()

eye_point at(xyz) up (xyz)

gluLookAt (ex ey ez 000 010)

draw_trianglerectanglepyramid etc

draw_axes

glutSwapBuffers () if double-buffered back buffer moves to front at retrace Also does a glFlush

glFlush() only needed if glutSwapBuffers is not used

copy DJ Foreman 2012 69

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

openGL 31

Using shaders amp Vertex Array ObjectsglBindVertexArray(my_vao_ID[0])

select 1st VAOglDrawArrays(GL_TRIANGLES 0 3) draw

1st object note difference from glBegin (GL_POLYGON)

hellip Note name of out(put) variable in

vertex shader must be the same as in(put) variable in fragment shader ie vertex -gt fragment (as in original

pipeline)copy DJ Foreman 2012 70

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

71

The Vertex Shader

Transform vertexs virtual 3D position to 2D screen coordinate

Manipulate position color and texture coordinate

Output to Geometry shader

generate new graphics primitives from those at beginning of pipeline

Rasterizer

copy DJ Foreman 2012

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Drawing

Last drawn object is in ldquoFRONTrdquo MUST set bit depth AND enable bit

depth test Rotation

Around a vector starting at 000 Use glRotate(Ѳ xyz)

Around any vector NOT anchored at 000Requires pre-multiplication of matrices with the

identity matrix

copy DJ Foreman 2012 72

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

73

Views

Orthographic view direction is perpendicular to projection

plane Matrix mode

Modelview Projection View

copy DJ Foreman 2012

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

74

The RenderMan Interface Bytestream

The RenderMan Interface is a standard interface between modeling programs and rendering programs capable of producing photorealistic quality images

httpwwwfaqsorgfaqsgraphicsrenderman-faq

httpwwwfaqsorgfaqsgraphicsrenderman-faqixzz0ggQ8591Y

RIB programs the target device (like PostScript does for printers)

copy DJ Foreman 2012

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

75

RIB files

The specification from Pixar httpsrendermanpixarcomproductsrispe

crispec_pdfRISpec3_2pdf

Course notes from SIGGRAPH conferences Siggraph 90 course 18 - The RenderMan

Interface amp Shading Language Siggraph 92 course 21 - Writing

RenderMan Shaders

copy DJ Foreman 2012

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Image Manipulation

Photographic images Gimp Photoshop etc

Modeling OpenGL ndash for creating and lighting a model modeling programs

Rendering converting a 3D (possibly wireframe) model to

a 2D image then lighting and (possibly) shading it

Ray tracing another way of lighting amp shading an image very CPU intensive

copy DJ Foreman 2012 76

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

77

Ray tracing Method for calculating the path of waves

particles through a system Two distinct forms

Ray tracing (physics) which is used for analyzing optical and other systems

Ray tracing (graphics) Used to generate a 3D image

Trace the path of light through pixels in an image plane Simulate the effects of striking virtual objects

Simulate reflection refraction scattering and chromatic (color) aberration

Compute intensive Ray-casting + follow reflections refractions amp

shadows

copy DJ Foreman 2012

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

78

Light terms

Absorption ndash unreflected+refracted light

Reflection ndash change of direction at an interface eg airsolid Angle = angle of incidence

Refraction ndash change of direction through media eg airwater Angle depends on refractive indices of both

media

copy DJ Foreman 2012

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

79

Ray tracing methodology Rays start at viewpoint go through viewplane

Eliminates unused rays from source Rays that do not strike objects Rays that are blocked by objects closer to viewer

Alternative (from light source) is photon mapping Each ray tested for intersections with objects Rays can be computed in parallel Algorithms

Estimate the incoming light at the intersection Examine the material properties of the object Combine above to calculate final color of the pixel

copy DJ Foreman 2012

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

80

Basic Ray tracing algorithm - 1 Initiate a ray (from viewpoint)

shade_pixel (original ray) This is a recursive call

(see next slide)

Apply ldquofinalrdquo shade to pixel Flat shading ndash assumes mono-colored object Cook-Torrance ndash uses physical attributes of

object Lambertian shading (cosine method)

Point brightness based on normal vector at point and vector from point to light source

httpwwwcsuncedu~rademachxroads-RTRTarticlehtml

copy DJ Foreman 2012

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

81

Ray tracing algorithm - 2

Shade_pixel (ray) Find 1st intersection C1= color of intersection point If reflective

Create reflection ray C2=Shade_pixel (new ray)

If transparent Create refraction ray C3=Shade_pixel (new ray)

Return combined color value (C1 C2 C3)

copy DJ Foreman 2012

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

82

Final coloring

shade = light_vector bull normal_vector if ( shade lt 0 )

shade = 0 point_color = object_color

( ambient_coefficient + diffuse_coefficient shade )

copy DJ Foreman 2012

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

83

Breeze DesignerAyam - lab Create a model of NCC1701 in BD

Notes The ldquofrontrdquo view is actually the back view Perspective view is not accurate

Save your model (it will be a CAD file) Use the ldquoprint screenrdquo key (copy to clipboard) Open RIB with Notepad amp print it Paste into Word amp print it Export your model as a RIB file Open RIB file (use ldquoopenrdquo not ldquoimportrdquo) in

Ayam Click on ldquoRunrdquo Use print screen again

copy DJ Foreman 2012

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

84

GimpGnu Image Manipulation Program

copy DJ Foreman 2012

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Operational concepts

3 windows Image ndash contains the actual results Tools ndash operators such as clone paint etc Layers ndash portions of the final image

Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied

independently of other changes

copy DJ Foreman 2012 85

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Gimp - lab

Load a digital landscape or cityscape photo Load a picture of yourself Select the image portion that is just yourself Extract the image of yourself ndash save it as a

JPG Insert it (inverted 90-180 deg) into the lsquoscape Save the new image as a new JPG file Send all 3 files (lsquoscape you merge) to the TA Grading

Image properly selected (20 pts) Image rotated (20 pts) Image properly overlayed (60 pts)

copy DJ Foreman 2012 86

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Video games vs Movies

Elements to manipulate Content Perspective Lightingshading Sequence Coloring

Video game images Dynamic

Movie images Pre-determined

copy DJ Foreman 2012 87

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

Alice

Developed at CMU

Language-free programming

environment

Point amp click usage

Rapid prototyping

copy DJ Foreman 2012 88

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

89

Alice environment

List of ldquoworldrdquo objects Insert into methods Modify characteristics (length width etc)

List of world details Properties Methods (user created) Functions (built-in methods)

World window (results) Method builder

copy DJ Foreman 2012

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

90

Alice - Programming

Programmer Selects ldquoverbsrdquo from list

For While Etc

Applies values for properties Statement structure pre-defined

Blocks pre-defined Function calls for ifhellip thenhellip else

No need to learn a ldquolanguagerdquocopy DJ Foreman 2012

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

91

Alice Lab

Open any of the Alice worlds Apply the following rules

1 Display 3 articulated characters (A B and C)

2 A flips onto its head and rotates slowly (5 times)

3 B waves both of its arms slowly (5 times)4 C walks around A and B while (2 and 3

happen)5 C changes direction and walks around A

and B once (while 2 and 3 repeat)6 Stop

Timing size shape position are your choice

copy DJ Foreman 2012

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

STEGANOGRAPHY and LSB analysis

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

93

What is it

Hiding a message in plain sight Inside another message Original is called the ldquocoverrdquo

Presence of message is not obvious Can be done with text or graphics

copy DJ Foreman 2012

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

94

How is it done

Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if

you know there is a hidden message you know the algorithm used

Can be very complex to detect amp extract

copy DJ Foreman 2012

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

95

LSB embedding ndash using MatlabFreemat

Concept Change the least significant bit of a set of

bytes 1rarr0 and 0rarr1 For all bytes

Repeat for multiple sets of bytes Example with grayscale images

One pixel is 8 bits Change the last bit Changes that pixelrsquos shade VERY slightly Repeat for every pixel

copy DJ Foreman 2012

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

96

LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)

c = imread(lsquomy_decoy_imagebmprsquo) Grayscale cover image lsquobrsquo is a vector of m bits (secret message)

k = 1 k is an index into the message bitsfor i = 1 height for j = 1 width LSB = mod(c[i j] 2) if kgtm | LSB = b[k] m is msg length in BITS s[i j] = c[i j] else

s[i j] = c[i j] + b[k] ndash LSB end k = k + 1 endend

imwrite(s lsquostego_imagebmprsquo lsquobmprsquo) write new image ldquosrdquo to

diskcopy DJ Foreman 2012

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

97

imread

The return value ldquocrdquo is an array containing the image data If the file contains a grayscale image ldquocrdquo is

an M-by-N array If the file contains a truecolor image ldquocrdquo is

an M-by-N-by-3 array The class of ldquocrdquo depends on the bits-per-

sample of the image data rounded to the next byte boundary Eg imread returns 24-bit color data as an

array of uint8 data because the sample size for each color component is 8 bits

copy DJ Foreman 2012

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

98

LSB extraction - algorithm

Extraction function Ext (Matlab syntax)s = imread(lsquostego_imagebmprsquo) Grayscale stego imagek = 1for i = 1 height for j = 1 width if k m how do I set the value of m b[k] = mod(s[i j] 2) k = k + 1 end endend

b is the extracted secret message as a bit string

copy DJ Foreman 2012

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

99

Properties of LSB flipping

LSBflip(x) = x + 1 ndash 2(x mod 2)

FlipLSB(x) is idempotent eg LSBflip(LSBflip(x)) = x for all x

LSB flipping induces a permutation on 0 hellip 255

0 1 2 3 4 5 hellip 254 255

LSB flipping is ldquoasymmetricalrdquo (eg 3 may change to 2 but never

to 4)

| LSB(x) ndash x | = 1 for all x (embedding distortion is 1 per pixel)

copy DJ Foreman 2012

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

100

Augmented Reality

Addition of computer-generated imagery to a real-world view

Differs from ldquomediated realityrdquo Real-world modified (+ or -)

Real-time Adds interactivity to real world view

ldquoComputer visionrdquo Systems that extract info from images

Medicalphotovideo scanners ldquoObject recognitionrdquo

copy DJ Foreman 2012

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

101

AR usage

Robotics Machining Assembly

Human interaction Auditory amp visual cueing Machine control

copy DJ Foreman 2012

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

102

AR developments

Heads-up displays Visual data projected onto a transparent screen

Windshield cockpit etc Used in auto and military ground- amp air-craft

Head-mounted displays Device with glasses goggles etc Also used in VR environments

Incorporated technology GPS Optical amp wireless sensors Gyroscopes Accelerometers

copy DJ Foreman 2012

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

103

Adobe Flash

(Originally Macromedia Flash) Vector graphics Uses

Video in web pages (swf) Movies (swf) Video games (swf) Standalone video (flv or exe)

Used with a flash player Telephones and other web-enabled devices

copy DJ Foreman 2012

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

104

Flash - 2

Support Adobe Flash CSn Adobe Integrated Runtime

Installs (silently) with Adobe Reader

Usage in a web pageltobject data=movieswfrdquo

type=applicationx-shockwave-flash width=500rdquo height=500gt ltparam name=movie value=movieswf gt

ltobjectgt

copy DJ Foreman 2012

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

105

Virtual Reality Modeling Language VRML

Text file format URLs may be associated with graphical

components May have JS or Java attached Called ldquoworldsrdquo File extension =wrl Many modeling programs can output VRML Supported by OpenVRML Succeeded by X3D

copy DJ Foreman 2012

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

106

X3D

Extensible 3D Graphics ISO standard XML-based file format Successor to VRML Applications which can parse X3D files

Project Wonderland Blender Freewrl

copy DJ Foreman 2012

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

107

X3d-Samplex3d

copy DJ Foreman 2012

ltxml version=10 encoding=UTF-8gtltDOCTYPE X3D PUBLIC ISOWeb3DDTD X3D 32EN

httpwwwweb3dorgspecificationsx3d-32dtdgtltX3D profile=Interchangeldquo version=32 xmlnsxsd=httpwwww3org2001XMLSchema-instance xsdnoNamespaceSchemaLocation= httpwwwweb3dorgspecificationsx3d-32xsd gtltScenegt lt-- viewable using freewrl --gt ltShapegt ltIndexedFaceSet coordIndex=0 1 2gt ltCoordinate point=0 0 0 1 0 0 05 1 0gt ltIndexedFaceSetgt ltShapegtltScenegtltX3Dgt

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

DIGITAL AUDIO

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

109

Terms

Level ndash power or volume dB ndash decibel - signal strength ratio

0 dB is the human hearing threshold Perception is proportional to log10 values of

level Breathing = 10dB A whisper = 30dB Lawnmower = 85dB Snowmobile = 105dB Jet engine (running) = 130 dB Rock concert = 110 ndash 140 dB

Hearing damage starts at 85 dB Pain starts at 130dB

copy DJ Foreman 2012

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

110

Basic editing

Effects Filter ndash specific freq Equalize - volume Modulate (speed reverb) Time amp pitch ndash stretch change pitch Noise reduction ndash breathing background

crackles Delay amp echo ndash room effects Clip - cut segments or limit

volumefrequency Copymovepaste - segments

copy DJ Foreman 2012

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

111

Multi-track editing-1

copy DJ Foreman 2012

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

112

Multi-track editing-2

Editing leftright channels of stereo signal

Comparison (eg forensics) Merging Appending Re-arranging

copy DJ Foreman 2012

MP3 MID

>

2004

Blues

11904381

>

Dont Call Me Now - Onyx Arranger demo songLRikhThis is a blues manner song created on basis of only one five-measureOOStyle - Count Basies Count On The Blues intro First piano trackand second Sax solo track were played by me on a MIDI keyboardand then processed using FX Chains Track 4 was created fromthe current chord sequences using Rendering command and thentransformed by a guitar accompaniment Modeler PMStyleOrchestrator settings Clips mixing Fills and Envelopes editingallowed achieving the desired song structureHave success with OnyxLeonid RikhNTONYXwwwjasminemusicnetDont Call Me Now - Onyx Arranger demo song

Chord SignsCm9Ab9G7Cm9Fm9Cm9Gm7G7Cm9F7sus2Cm7Ebmaj9Ab6AbmajFm7Bb6Cm9Cm9x7G7Fm9Cm9G7Ab7Cm9Fm9Cm9Gm7G7G7Cm9F7sus2Cm7Fm9Cm9G7Ab7Cm9Ab9G7FEbmaj9

KeyScaleCMinorCMinorCMinorCMinorCMinor

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

113

Music creation

Midi ndash Musical Instrument Digital Interface Specifies musical instructions

Instrument to play Notes durations volumes Binary file much smaller than MP3 Two ways

1 From instrument Instrument connects to PC PC software creates filesheet music

2 From written instruction Use a GUI to define notes positions

copy DJ Foreman 2012

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

114

copy DJ Foreman 2012

DIGITAL VIDEO

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

115

TV Formats

Digital (ATSC) Over the Air (OTA) ndash full HD (1080P) Scrambled cable ndash full HD (1080P) standard definition (SD=480i) QAM ndash NOT scrambled ndash 720P to 1080P

Cable-box NOT required Includes ATSC and converted NTSC Limited availabilty

Analog (NTSC) Unscrambled cable (limited availability) VCR output Only in SD

copy DJ Foreman 2012

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

116

Recording

Digital Cameras SD card files - AVI format DVD ndashMPG-2 or -4 Tape ndash DV format requires real-time transfer

Digital TV tuners Receives pre-encoded (OTAcable) MPG-2 to

PC Eg SiliconDust HD Homerun

Analog TV tuners Receives NTSC QAM amp cableVCR analog Does hardware encoding to MPG-2

Eg Hauppauge WinTV HVR 2250 also does OTA digital

copy DJ Foreman 2012

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

117

Remote Playback Hardware

Boxee - pre-recorded formats + internet WDTV Live ndash pre-recorded formats + internet Sage TV ndash pre-recorded formats + Tuner scheduler

Supports only SageTV streaming receiver Bought by Google removed from market

Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)

Xbox ndash WMV only PS3 - MPG-2 only Wii ndash flash only using Opera browser

Software Tversity

httpwwwmaximumpccomarticlestreamingpage=01 WMC ndashWindows 7 (gt=Home Premium)+ Xbox SageTV Hauppauge WinTV

copy DJ Foreman 2012

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

118

copy DJ Foreman 2012

DV EDITING

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

119

Terms Asset list - Collection of clips frames amp stills Preview

Editable version of show Allows playback control + editing

Timeline - Time-marks abovebelow path of frames

Footage ndash DV segments measured in time units Effects - Re-size crop tone color super-

imposing Transitions

Straight cut (no transition) Fade inout dissolve cross-dissolve etc

copy DJ Foreman 2012

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

120

DV Editing -1

Import clips frames stills Trim footage (recurring action)

Insertremove segments Re-arrange segments Does not alter original file

Linear edit Simple re-arranging insertion removal

Non-linear edit (next slide) Note next slide compare ALL edits to

original

copy DJ Foreman 2012

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

121

Non-linear editing original

Ripple (middle expands 1)

Rolling (middle expands 1) (following In changed)

Slip (middle InOut change)

Slide (Middle moves left 2) (Prev Out next In

change)

DV Editing - 2

copy DJ Foreman 2012

a cb d 4 5 76 C FED G

a cb d 4 5 76 C FED G8

a cb d 4 5 76 FED G8

a cb d 1 2 3 C FED G4

a b 4 5 76 A DCB E F G

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

122

DV Editing -3

Ripple ndash changes overall length Following edits maintain overall length

Rolling ndash shortenslengthens following clip by amount of insertiondeletion

Slip ndash only the edited clips In and Out points change

Slide ndashchanges prev Out next In

copy DJ Foreman 2012

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

123

DV formats VCD

VHS quality Plays anywhere 7480 minutes recording for 650700MB CDs

SVCD 3560 minutes Many standalone DVD players all CDDVD-ROM

CVD = lower resolution less noise than SVCD DVD

2 hrs of good quality video (437 GB)+ DolbyDTS audio

Blu-ray High quality video

copy DJ Foreman 2012

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

124

DVD types Singledouble-sided 437GB 875GB

DVDForum DVD ndash R non-rewritable DVD ndash RW re-writable

DVD+RW Alliance DVD + R non-rewritable DVD + RW re-writeable

DVD plusmn R DL Dual layer 159GB

DVD-RAM NOT player compatible asymp HDD

+ and ndash differences are old hw designscopy DJ Foreman 2012

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2

125

DVD types -2

Blu-ray 25 GB per layer up to 2 layers

AVCHD (Panasonic SONY) For Hi-DEF camcorders

HD-DVD (MS Toshiba NEC Sanyo) discontinued in 2008

VHS (everyone) vs Betamax (SONY)

copy DJ Foreman 2012

  • CS338
  • Overview -1
  • Topics (general order of coverage)
  • Software we might use (all free)
  • Image formats
  • Bitmap formats
  • Vector formats
  • Compression
  • Codecs
  • Codecs - 2
  • Codec References
  • Considerations
  • Lossless Encoding
  • Methodology
  • Huffman lab ndash more details
  • Sample of prefix coding amp compression
  • Compression-Model types
  • Standard Compressed-File Content
  • Example 1 Header file for AU files
  • Audio Encoding Enumerations
  • Example 2 ZIP files (PkWare)
  • More info on encoding
  • Recording Bit-Rate
  • Lab assignment ndash 1 ndash part 1
  • Lab assignment 1 ndash part 2
  • Working with Images
  • Containers
  • Container file format (MP4QT)
  • Pixels
  • Image creation
  • Photography (film)
  • FYI ndash a little chemistry
  • Resolution
  • Photography (digital)
  • Some Comparisons
  • Working Definitions - 1
  • Working Definitions - 2
  • Renderman Interface Bytestream (RIB) copy Pixar
  • Context
  • OpenGL 20 Fixed Function Pipeline
  • The Vertex Processor
  • OpenGL 20 Vertex Processor Pipeline
  • DirectX vs OpenGL
  • The Model-view Matrix
  • Graphic fragment
  • Fragment Processing
  • Lab 2- basic modeling
  • Lab 2 grading
  • Some Examples for lab 2
  • OpenGL
  • Window Control
  • Programming in OpenGL
  • OpenGL Program Structure
  • Views
  • Image Rotation
  • Program structure
  • GlutReshapeFunc
  • GL_Load_Identity
  • gluLookAt
  • Matrix Modes
  • Keyboard-related functions
  • Basic Functions in ldquomainrdquo
  • Skeleton of program ndash slide 1
  • Skeleton of program ndash slide 2
  • Skeleton of program ndash slide 3
  • Initialize the program ndash slide 4
  • Create a colored line ndash slide 5
  • Skeleton of program ndash slide 6
  • Basic Display Function
  • openGL 31
  • The Vertex Shader
  • Drawing
  • Views (2)
  • The RenderMan Interface Bytestream
  • RIB files
  • Image Manipulation
  • Ray tracing
  • Light terms
  • Ray tracing methodology
  • Basic Ray tracing algorithm - 1
  • Ray tracing algorithm - 2
  • Final coloring
  • Breeze DesignerAyam - lab
  • Gimp Gnu Image Manipulation Program
  • Operational concepts
  • Gimp - lab
  • Video games vs Movies
  • Alice
  • Alice environment
  • Alice - Programming
  • Alice Lab
  • Steganography and LSB analysis
  • What is it
  • How is it done
  • LSB embedding ndash using MatlabFreemat
  • LSB embedding - algorithm
  • imread
  • LSB extraction - algorithm
  • Properties of LSB flipping
  • Augmented Reality
  • AR usage
  • AR developments
  • Adobe Flash
  • Flash - 2
  • Virtual Reality Modeling Language
  • X3D
  • X3d-Samplex3d
  • Digital audio
  • Terms
  • Basic editing
  • Multi-track editing-1
  • Multi-track editing-2
  • Music creation
  • Digital Video
  • TV Formats
  • Recording
  • Remote Playback
  • DV Editing
  • Terms
  • DV Editing -1
  • DV Editing - 2
  • DV Editing -3
  • DV formats
  • DVD types
  • DVD types -2