"In the country of the blind, the one-eyed man is king"

Visa-Valtteri “visy” Pimiä

or how fumbling my way through pixel-based effects made learning WebGL a lot easier

My history with pixel-based visuals

Started doing simple effects with QBASICWorked my way to Turbo Pascal and VGA programmingGot sidetracked to NES and Atari 2600 developmentLearned OpenGL mostly professionally, then started using GPU-accelerated graphics

But why talk about something whenI can just speedcode here :-)

Processing makes it possible

Coding pixel-based effects

Access the screen through a framebuffer, an array of pixel color values, palette index values or bitplane togglesSet pixel color values according to position, time, function value etc.

Proprietary “engines”

I started making small games and demo-style little programsEverything was always coded for one single production, with no mind put into reusability of the toolsThis got old pretty quickly and I found myself toying with the idea of a visual editor for demos

First: S.P.L.I.F

Trilobit ScriPtable Lua Interactive MultiplatForm Tackles a multitude of problems I had with development:

Inherently multiplatform (OSX, Win32, Linux) because all the GL calls are coded with Lua.All visual code is reloadable without relaunching or recompiling the programThe demo player is seekable and supports setting sync points “The demo is the editor”Always open source

Syncing

Timeline file millisecond syncing of demo partsFFT beat sync for effectsSeeking for testing

S.P.L.I.F has flaws

Fixed OpenGL version, upgrading would require recompiling the Lua libraries for all the three platformsNo shader supportTerrible hacks for texture handlingThe OpenGL / Lua bound version I used was found to be rather incomplete

But it worked well enough

Counter Intelligence Program by Bilotrip

Need for an upgrade

After I heard about WebGL I got the idea of doing something like S.P.L.I.F in the browser with easily reloadable and tweakable visualsWebGL has shaders, so the natural choice was to enable the editing of shader parameters (in addition for JavaScript variables) realtime

Technology behind DarumaGL

WebGL canvas for the outputPure JavaScript + HTML5 + CSS3 for the UITextures loaded from PNG filesHTML5 audio from OGG fileTimeline.js used for the keyframe editor

Demonstration of the DarumaGL editor

http://www.low.fi/~visy/webgl/iloinen_paiva/

…and it has flaws as well

No real timeline dumping (cut & paste from export windows)Shader code still requires reloading of the pageHTML5 audio support is still rather flaky cross browsers, and require some time to matureNo cross-browser FFT support for sound analysisWebGL is still not available for mobile browsers or Internet Explorer

But…

WebGL _is_ a W3 standard, and pretty much all major browser except Internet Explorer have support for it in development roadmapsNo plugins required anymore for GPU accelerated content and complex video/audio data processing and playback

No third party components required

Lessons learned from pixel effects

Pixel shaders are pretty similar to framebuffers in many waysCan set single resolution independent fragments instead of toggling actual pixels on the screenMany fragments are calculated concurrently on the GPU, so the shader program is run in parallel.

Some things of note

I tend to avoid calculating complex geometries and like to play around with fragments on a single quadComplex blending and processing passes possible with the GPU -> enables effects that are just simply impossible to do on a framebuffer with a fast FPS

Demonstration

MKULTRA by Bilotriphttp://www.low.fi/~visy/webgl/mkultra/

Thank you for listening!