Image Filtering AdvancedImage Filtering Advanced

Image filtering with GDI and DX HW accelerations

Image Filtering Advanced

An introduction : domain terms

An introduction : domain terms

♦ Bitmap (let’s assume Bitmap == GDI+ Bitmap)System memory portion containing Pixel as color’s byte

♦ DX SurfaceSystem or Video memory portion representing a bitmap

♦ TextureMainly Video memory portion containing a bitmap that

will be mapped onto a polygon

Image Filtering Advanced

Enhanced Image Filter Library

Enhanced Image Filter Library

♦ What does a filter do?

♦ What is a filter?

♦ What is a filter chain?

A filter is a function from image to image,

A filter chain is a functional composition so, it’s a function again.

Image Filtering Advanced

The speed part 1 : unsafeThe speed part 1 : unsafeprotected override void filter(Bitmap b, Bitmap[] temp) {

BitmapData bmData = b.LockBits(new Rectangle(0, 0, b.Width, b.Height), ImageLockMode.ReadWrite, PixelFormat.Format24bppRgb); int stride = bmData.Stride; System.IntPtr Scan0 = bmData.Scan0;

unsafe {byte * p = (byte *)(void *)Scan0;

int nOffset = stride - b.Width*3;int nWidth = b.Width * 3;

for(int y=0;y<b.Height;++y) { for(int x=0; x < nWidth; ++x ) {

p[0] = (byte)(255-p[0]);++p;

} p += nOffset;}

} b.UnlockBits(bmData);}

}

This way is faster than accessing pixels through the GetPixel GDI function

Image Filtering Advanced

Filter

Brightness

ColorCorrector

Contrast

Conv3x3

EdgeDetectConvolution EdgeDetectDifference

EdgeDetectHomogenity

EdgeDetectHorizontal

EdgeDetectQuick

EdgeDetectVertical

EdgeEnhance EmbossLaplacian

Gamma

GaussianBlur

GrayScale

Invert

MeanRemoval

Sharpen

Smooth

Library structureLibrary structure

Image Filtering Advanced

The key for success : the filter base class design

The key for success : the filter base class design

[Serializable] public abstract class Filter {

private Filter next;protected Filter(Filter f) { next = f;}protected Filter() { next = null;}protected abstract void filter(Bitmap b, Bitmap[] temp);

protected abstract int NumCopies { get; }

public BaseConf GetConfig(){ return new BaseConf();}

public void Apply(Bitmap b) { int n = this.NumCopies; Filter p = next; while (p != null) {

n = Math.Max(n, p.NumCopies);p = p.next;

} Bitmap[] tmp = new Bitmap[n]; for (int i = 0; i < n; i++)

tmp[i] = new Bitmap(b.Width, b.Height); p = this; do {

p.filter(b, tmp);p = p.next;

} while (p != null); foreach (Bitmap bm in tmp) {

bm.Dispose(); }}

}

Create all the temporaryImages before your filterchain goes with its work!

Image Filtering Advanced

The speed part 2 : ngenThe speed part 2 : ngen

♦ Let’s get native!

Run the ngen tool on the filter library assembly to get it compiled once for all.

Image Filtering Advanced

♦ A little trick to achieve the real magic

♦ Let’s pretend :

♦ Remember you gain speed but you get also limitations!

Image => plane => polygon

The speed part 3 : Get HW!The speed part 3 : Get HW!

Image Filtering Advanced

DX StructureDX Structure

♦ We can use the HW 3D acceleration to boost 2D image filtering using the Programmable Pixel Shader

Image Filtering Advanced

HSL with DX 9HSL with DX 9

♦ Cross Hardware language!

♦ Can contains multiple techniques and passes!

♦ Exposes a large number of common function as lerp and so on

Image Filtering Advanced

HSL Stupid ExampleHSL Stupid Example

float4 Light(float3 LightDir : TEXCOORD1, uniform float4 LightColor, float2 texcrd : TEXCOORD0, uniform sampler samp) : COLOR

{

float3 Normal = tex2D(samp,texcrd);

return dot((Normal*2 - 1), LightDir)*LightColor;

}

Image Filtering Advanced

The IdeaThe Idea

1. Create a DX device on a control

2. Create a 3D plane

3. Put the image you want to process on the plane as a texture

4. Use the Pixel Shader Program to make the GPU works for you

5. Use the Control Graphics class to save your processed image.

Image Filtering Advanced

A simple but real FX example

A simple but real FX example

Image Filtering Advanced

And the cons?And the cons?

Here we are . . .

♦ You can use only power of 2 sized bitmap

♦ The display size is the maximum size for the output image

. . . But you get a common PC running as it’s got a real expensive (and dedicated) DSP!

Image Filtering Advanced

The Direct Show wayThe Direct Show way

♦ DShow use a graph from sources to renderes (audio and video)

♦ Along the graph you can have filters

♦ 2 approaches – In place transformations– Not in place transformations

♦ Only C++ (the unmanaged one!)

Image Filtering Advanced

CTransformFilter::Transform

CTransformFilter::Transform

♦ This filter uses the CTransformInputPin class for its input pin, and the CTransformOutputPin class for its output pin.

♦ Use this base class if you want to try filter and then pass it through the graph

♦ Beware of memory leak!

Image Filtering Advanced

Example of transformExample of transformHRESULT CRleFilter::Transform(IMediaSample *pSource, IMediaSample *pDest) {

// Get pointers to the underlying buffers. BYTE *pBufferIn, *pBufferOut; hr = pSource->GetPointer(&pBufferIn); if (FAILED(hr)) {

return hr; } hr = pDest->GetPointer(&pBufferOut); if (FAILED(hr)) {

return hr; } // Process the data. DWORD cbDest = EncodeFrame(pBufferIn, pBufferOut); KASSERT((long)cbDest <= pDest->GetSize()); pDest->SetActualDataLength(cbDest); pDest->SetSyncPoint(TRUE); return S_OK;

}

Image Filtering Advanced

CTransInPlaceFilter::Transform

CTransInPlaceFilter::Transform

♦ Transform the input sample in place

♦ Use it for real real-time processing

♦ What do you want more?

Image Filtering Advanced

The Frame Dispatcer ClassThe Frame Dispatcer Class

If you want to apply filter to a real time camera or to an avi file in C# there’s an utility from the Medialab (university of Pisa) that can help you to get your frames inside a managed application!

Image Filtering Advanced

ReferencesReferences

♦ Enhanced Filter Library and Frame Dispatcer– http://dotnet.di.unipi.it