1

Extended SQL for Multimedia Queries

select <whats> from <whatr>where <preds>orderby <whato)groupby <whatg>

<whats> can be alphanum, images, functions<functions> can be numeric-valued, vector- valued (such as histograms), image-valued (such as extractors), aggregates,….<preds> can be hard constraints or similarity functions<whato) can be alphanum or similarities<whatg> can be alphanum or similarities

2

Queries from Current System

select A.id, A.img, simfMRI(A.img, B.img)from fMRI A, fMRI Bwhere B.id = 1 orderby simfMRI(A.img,B.img)

select A.id, I.img, S.probfrom fMRI A, simfMRI Swhere S.img = A.imgorderby S.prob

fMRI simfMRIid img img prob

*Assumes simfMRI is a global comparison of 2 fMRI images.

similarities of all images to aselected one withid = 1

3

Queries from Current System

select A.id, A.imgfrom fMRI A, fMRI Bwhere B.id = 1 and simfMRI(A.img, B.img)

select distinct A.id, A.imgfrom fMRI A, simfMRI Swhere S.fid = A.fid

fMRI simfMRIfid id img img prob

*Assumes simfMRI is a global comparison of 2 fMRI images.

similarities of all images to aselected one withid = 1

User query:

MystiQ query:

4

Potential Queries from Very Simple and Beyond

select simfMRI(A.img,B.img)from fMRI A, fMRI Bwhere A.id = 12 and B.id = 37

Return the similarity between the fMRI images ofpatient 12 and patient 37.

fMRI simfMRIid img prob single similarity value

5

Find pairs of patients in the database having the same age and similar fMRI images.

select A.id, B.id, P.age, simfMRI(A.img, B.img)from patient P, patient Q, fMRI A, fMRI Bwhere P.ID = A.id and Q.ID = B.id and A.age = B.ageorderby simfMRI(A.img, B.img)

patient fMRI simfMRIid age id img img img prob

pairs of database images inorder of similarity

Q3

6

Find pairs of patients in the database having the same age and similar fMRI images.

select A.id, B.id, P.age, from patient P, patient Q, fMRI A, fMRI Bwhere P.ID = A.id and Q.ID = B.id and A.age = B.age and simfMRI(A.img, B.img)

patient fMRI simfMRIid age fid1 fid2 prob

pairs of databaseimages

Q3

fid id img

User query:

select distinct A.id, B.id, P.age, from patient P, patient Q, fMRI A, fMRI B, simfMRIwhere P.ID = A.id and Q.ID = B.id and A.age = B.age and simfMRI.fid1 = A.fid and simfMRI.fid2=B.fid

MystiQ query:

Note: twopatients A.id,B.id may havemultiple similarimages. Theprobability is non-trivial.

7

Given a query image called thisimg, return the id’sand similarities for database images that are similarenough.

select A.id, P.age, A.img, simfMRI(thisimg, A.img)where P.ID = A.id and simfMRI(thisimg,A.img) > threshorderby simfMRI(thisimg, A.img)

thisimg thresh patient fMRI simfMRIid age id img img prob

Q4

8

Given a query image called thisimg, return the id’sand similarities for database images that are similarenough. select A.id, P.age, A.img

from patient P, fMRI Awhere P.ID = A.id and simfMRI(thisimg,A.img) > thresh and simfMRI(thisimg, A.img)

thisimg patient fMRI simfMRI

id age

Q4

User query:

MystiQ query:fid probfid id imgimg

select A.id, P.age, A.img, simfMRI(thisimg, A.img)from patient P, fMRI A, simFMRI Swhere P.ID = A.id and simfMRI(thisimg,A.img) > thresh and A.fid = S.fid

9

Given a query image called thisimg, return the id’sand similarities for database images that are similarenough.

select A.id, P.age, A.imgfrom patient P, fMRI Awhere P.ID = A.id and and HIGH(simfMRI(thisimg,A.img)) [weight=7] and simfMRI(thisimg, A.img) [weight=0]

thisimg patient fMRI simfMRI

id age

Q4

Better user query:

fid probfid id imgimg

10

Weights

I’m proposing extending the probabilistic predicatesand/or probabilistic tables with weightsE.g. FROM simfMRI S [weight=3.5]WHERE simfMRI(img1,img2) [weight=-0.5]

Default weight = 0.0Positive weight means the predicate is more importantNegative weight means it is less important.

11

High/Low Predicates

I’m also proposing introducing two newprobabilistic predicates:

High(value) = more “true” the higher “value” is

Low(value) = more “true” the lower “value” is

These predicates make sense only within a query:Different values are compared and the higher (lower)receive higher probability.

12

Given an image and a location, return id’s, images, and activation levels from database images that have high enough activation levels at that location in order bysimilarity at that particular location.

select A.id, A.img, activation(A.img,loc), simsignal(signal(A.img,loc), signal(thisimg,loc))from fMRI Awhere activation(A.img,loc) > threshorderby simsignal(signal(A.img,loc), signal(thisimg,loc)

thisimg loc thresh fMRI simsignal activation signal

signal prob(, ) (,)

activation is a function that returns the mean activation level at a given location.signal is a function that returns the signal at a given location.locations denote particular voxels.

Q5

fid id img

13

Given an image and a location, return id’s, images, and activation levels from database images that have high enough activation levels at that location in order bysimilarity at that particular location.

select A.id, A.img, activation(A.img,loc)from fMRI A, thisimg Twhere activation(A.img,loc) > thresh and simsignal(signal(A.img,loc), signal(T,loc))

thisimg loc thresh fMRI simsignal activation signal

signal prob (, ) (,)

activation is a function that returns the mean activation level at a given location.signal is a function that returns the signal at a given location.locations denote particular voxels.

Q5

User query:

fid id img

14

Given an image and a location, return id’s, images, and activation levels from database images that have high enough activation levels at that location in order bysimilarity at that particular location.

select A.id, A.img, activation(A.img,loc)from fMRI A, thisimg Twhere high(activation(A.img,loc)) [weight=10.0] and simsignal(signal(A.img,loc), signal(T,loc))

thisimg loc thresh fMRI simsignal activation signal

signal prob(, ) (,)

activation is a function that returns the mean activation level at a given location.signal is a function that returns the signal at a given location.locations denote particular voxels.

Q5

Better user query:

fid id img

15

Return ids and images for images (A) that have thehighest activation level of all database images (B) at a given location. For each of these, return id’s andimages (C) of all database images whose activationis above a threshold at that location, ordered bytheir signal similarity to A at that location.

select A.id, A.img, C.id, C.imgfrom fMRI A, fMRI B, fMRI Cwhere activation(A.img,loc) = max(activation(B.img,loc)) and activation(C.img,loc) > threshorderby simsignal(signal(A.img,loc),signal(C.img,loc))

loc thresh fMRI simsignal activation signalid img signal prob (, ) (,)

Q6

16

Return ids and images for images (A) that have thehighest activation level of all database images at a given location. For each of these, return id’s andimages (C) of all database images whose activationis above a threshold at that location, ordered bytheir signal similarity to A at that location.

select A.id, A.img, C.id, C.imgfrom fMRI A, fMRI Cwhere high(activation(A.img,loc)) [weight = 100.0]and high(activation(C.img,loc)) [weight = 0.0]and simsignal(signal(A.img,loc),signal(C.img,loc)) [weight=0.0]

loc thresh fMRI simsignal activation signalsignal prob (, ) (,)

Q6

User query:

fid id img

17

Extract and return a neighborhood of a given size abouta given location from a given query image.

select nbd(thisimg, loc, size)from thisimg

thisimg loc size

Simple example of an extraction function.

Q7

18

select A.id, A.img, I.loc, simnbd(nbd(A.img, I.loc, size), nbd(thisimg, thisloc, size)) as Sfrom fMRI A, intloc Iwhere A.img = I.img orderby S

Return id’s, images, and locations for images that havesimilar neighborhoods of a given size at that locationto neighborhoods of the query image thisimg at locationthisloc of the same given size.

thisimg thisloc fMRI intloc simnbd

img loc img nbd prob

The intloc relation contains the locations of interesting voxels in an image.This is not straightforward, as we are computing similarity of neighborhoods of images, which are obtained through function calls, not through relations.

thisimg

A

Q8

id img

19

select A.id, A.img, I.locfrom fMRI A, intloc Iwhere A.fid = I.fid and simnbd(nbd(A.img, I.loc, size), nbd(thisimg, thisloc, size))

Return id’s, images, and locations for images that havesimilar neighborhoods of a given size at that locationto neighborhoods of the query image thisimg at locationthisloc of the same given size.

thisimg thisloc fMRI intloc simnbd

fid loc prob img nbd prob

The intloc relation contains the locations of interesting voxels in an image.This is not straightforward, as we are computing similarity of neighborhoods of images, which are obtained through function calls, not through relations.

thisimg

A

Q8

User query:

fid id img

20

select A.id,A.img, I.locfrom fMRI A, SUR S, intloc I where A.img = I.img and simloc(I.loc, S.loc)

Return id’s, images and pattern locations of database imagesthat have an interesting pattern at a location that is similar toone of the locations of language sites in the samepatient’s SUR data.

fMRI SUR intloc simlocid img id loc img loc loc loc prob

Q9

21

select A.id,A.img, I.locfrom fMRI A, SUR S, intloc I where l.fid = A.fid and simloc(I.loc, S.loc)

Return id’s, images and pattern locations of database imagesthat have an interesting pattern at a location that is similar toone of the locations of language sites in the samepatient’s SUR data.

fMRI SUR intloc simlocid img id loc loc loc prob

Q9

User query:

fid loc prob

22

select A.id,A.img, I.loc, S.locfrom fMRI A, SUR S, intloc I where A.img = I.img and A.id = S.id and simloc(I.loc, S.loc) and simpeaks(signal(A.img,I.loc),S.signal)

Return id’s, images and pattern locations of database imagesthat have an interesting pattern at a location that is similar toone of the locations of language sites in the samepatient’s SUR data and whose fMRI signal at the fMRI locationhas a similar peak pattern to the SUR signal at its location.

fMRI SUR intloc simloc simpeaks

id img id loc signal img loc loc loc prob signal signal prob

Q10 We don’t understand this query yet