Post on 07-Nov-2014
description
"In the country of the blind, the one-eyed man is king"
Visa-Valtteri “visy” Pimiä
or how fumbling my way through pixel-based effects made learning WebGL a lot easier
My history with pixel-based visuals
Started doing simple effects with QBASICWorked my way to Turbo Pascal and VGA programmingGot sidetracked to NES and Atari 2600 developmentLearned OpenGL mostly professionally, then started using GPU-accelerated graphics
But why talk about something whenI can just speedcode here :-)
Processing makes it possible
Coding pixel-based effects
Access the screen through a framebuffer, an array of pixel color values, palette index values or bitplane togglesSet pixel color values according to position, time, function value etc.
Proprietary “engines”
I started making small games and demo-style little programsEverything was always coded for one single production, with no mind put into reusability of the toolsThis got old pretty quickly and I found myself toying with the idea of a visual editor for demos
First: S.P.L.I.F
Trilobit ScriPtable Lua Interactive MultiplatForm Tackles a multitude of problems I had with development:
Inherently multiplatform (OSX, Win32, Linux) because all the GL calls are coded with Lua.All visual code is reloadable without relaunching or recompiling the programThe demo player is seekable and supports setting sync points “The demo is the editor”Always open source
Syncing
Timeline file millisecond syncing of demo partsFFT beat sync for effectsSeeking for testing
S.P.L.I.F has flaws
Fixed OpenGL version, upgrading would require recompiling the Lua libraries for all the three platformsNo shader supportTerrible hacks for texture handlingThe OpenGL / Lua bound version I used was found to be rather incomplete
But it worked well enough
Counter Intelligence Program by Bilotrip
Need for an upgrade
After I heard about WebGL I got the idea of doing something like S.P.L.I.F in the browser with easily reloadable and tweakable visualsWebGL has shaders, so the natural choice was to enable the editing of shader parameters (in addition for JavaScript variables) realtime
Technology behind DarumaGL
WebGL canvas for the outputPure JavaScript + HTML5 + CSS3 for the UITextures loaded from PNG filesHTML5 audio from OGG fileTimeline.js used for the keyframe editor
Demonstration of the DarumaGL editor
http://www.low.fi/~visy/webgl/iloinen_paiva/
…and it has flaws as well
No real timeline dumping (cut & paste from export windows)Shader code still requires reloading of the pageHTML5 audio support is still rather flaky cross browsers, and require some time to matureNo cross-browser FFT support for sound analysisWebGL is still not available for mobile browsers or Internet Explorer
But…
WebGL _is_ a W3 standard, and pretty much all major browser except Internet Explorer have support for it in development roadmapsNo plugins required anymore for GPU accelerated content and complex video/audio data processing and playback
No third party components required
Lessons learned from pixel effects
Pixel shaders are pretty similar to framebuffers in many waysCan set single resolution independent fragments instead of toggling actual pixels on the screenMany fragments are calculated concurrently on the GPU, so the shader program is run in parallel.
Some things of note
I tend to avoid calculating complex geometries and like to play around with fragments on a single quadComplex blending and processing passes possible with the GPU -> enables effects that are just simply impossible to do on a framebuffer with a fast FPS
Demonstration
MKULTRA by Bilotriphttp://www.low.fi/~visy/webgl/mkultra/
Thank you for listening!