Data Mining on Streams

We should use runlists for stream data mining (unless there is some spatial structure to the data, of course, then we need to use spatially oriented techniques) since a runlist can be truncated at one end and appended to at the other very easily (a Ptree, even a 1-D Ptree cannot accommodate such activity gracefully. However, if the data is spatial and there is a need for the continuity advantage of 2-D Ptrees, then Ptrees should be used!).

We begin with some slides reviewing Ptrees, RunLists and etc. Then move to stream Data Mining.

6. 1st half of 1st of 2nd is 1

00 0 0 1 1

4. 1st half of 2nd half not 0 00 0 0

2. 1st half is not pure1 0

00 0

1. Whole file is not pure1 0

Horizontal structureProcessed vertically(scans)

P11 P12 P13 P21 P22 P23 P31 P32 P33 P41 P42 P43 0 0 0 0 1 10

0 1 0 0 1 01

0 0 00 0 0 1 01 10

0 1 0

0 1 0 1 0

0 0 01 0 01

0 1 0

0 0 0 1 0

0 0 10 1

0 0 10 1 01

0 0 00 1 01

0 0 0 0 1 0 010 015. 2nd half of 2nd half is 1

00 0 0 1

00001011

horizontally process these Ptrees using one multi-operand logical AND operation.

Ptrees vertical partition; compress each vertical bit slice into a basic Ptree;

010 111 110 001011 111 110 000010 110 101 001010 111 101 111101 010 001 100010 010 001 101111 000 001 100111 000 001 100

R( A1 A2 A3 A4)

A table, R(A1..An), is a horizontalstructure (set of horizontal records)processed vertically (vertical scans)

1-Dimensional Ptrees are built by recording the truth of the predicate “pure 1” recursively on halves, until there is purity, P11:

3. 2nd half is not pure1 0 00 0

7. 2nd half of 1st of 2nd not 0

00 0 0 1 10

010 111 110 001011 111 110 000010 110 101 001010 111 101 111101 010 001 100010 010 001 101111 000 001 100111 000 001 100

--> R[A1] R[A2] R[A3] R[A4]

0 1 0 1 1 1 1 1 0 0 0 10 1 1 1 1 1 1 1 0 0 0 00 1 0 1 1 0 1 0 1 0 0 10 1 0 1 1 1 1 0 1 1 1 11 0 1 0 1 0 0 0 1 1 0 00 1 0 0 1 0 0 0 1 1 0 11 1 1 0 0 0 0 0 1 1 0 01 1 1 0 0 0 0 0 1 1 0 0

R11 R12 R13 R21 R22 R23 R31 R32 R33 R41 R42 R43

Eg, to count, 111 000 001 100s, use “pure111000001100”: 0 23-level

P11^P12^P13^P’21^P’22^P’23^P’31^P’32^P33^P41^P’42^P’43 = 0 0 22-level =201 21-level

1. 1st run is Pure0 0:000 truth:start

010 111 110 001011 111 110 000010 110 101 001010 111 101 111101 010 001 100010 010 001 101111 000 001 100111 000 001 100

R( A1 A2 A3 A4)

Run Lists: Another way to handle vertical data. Generalized Ptrees using standard run length compression of vertical bit files (alternatively, using Lempl Zipf?, Golomb?, other?)

Run Lists: record the type and start-offset of pure runs. E.g., RL11:

010 111 110 001011 111 110 000010 110 101 001010 111 101 111101 010 001 100010 010 001 101111 000 001 100111 000 001 100

--> R[A1] R[A2] R[A3] R[A4]

0 1 0 1 1 1 1 1 0 0 0 10 1 1 1 1 1 1 1 0 0 0 00 1 0 1 1 0 1 0 1 0 0 10 1 0 1 1 1 1 0 1 1 1 11 0 1 0 1 0 0 0 1 1 0 00 1 0 0 1 0 0 0 1 1 0 11 1 1 0 0 0 0 0 1 1 0 01 1 1 0 0 0 0 0 1 1 0 0

R11 R12 R13 R21 R22 R23 R31 R32 R33 R41 R42 R43

2. 2nd run is Pure1 1:100

3. 3rd run is Pure0 0:101

4. 4th run is Pure1 1:110

RL11 0:000

1:1000:1011:110 (to complement, flip purity bits)

Eg, to count, 111 000 001 100s, use “pure111000001100”: RL11^RL12^RL13^RL’21^RL’22^RL’23^RL’31^RL’32^RL33^RL41^RL’42^RL’43

RL11 RL12 RL13 RL21 RL22 RL23 RL31 RL32 RL33 RL41 RL42 RL43 0:0001:1000:1011:110

1:0000:1001:101

0:0001:0010:0101:1000:1011:110

1:0000:100

1:0000:110

1:0000:0101:0110:100

1:0000:100

1:0000:010

0:0001:010

0:0001:010

0:0001:0100:101

1:0000:0011:0100:1001:1010:110

00001011

R11

Other Indexes on RunListsWe could put Pure0-Run, Pure1-Run and Mixed-Run Indexes on RLs:

0000101101010101

R11

RL11 00:0 11:100 00:101 11:110 01:1000

1RL11 0100:1 0110:2

0RL11 0000:4 0101:1

start length

MRL11 1000:8

Or since we would not traverse the RL very often make it a link list and just concat indexes

0RL11 0000:4 0101:1

MRL11 1000:8

1RL11 0100:1 0110:2

START

Indexed RunLists ANDing

0000101101010101

R11

1RL11 0100:1 0110:2

0RL11 0000:4 0101:1

MRL11 1000:8:01011111

0010100001011111

R34

RL11

1RL34 1011:5

0RL34 0000:2 0101:4

MRL34 0010:3:101 1001:2:10

RL34

1RL11^34 0100:1

0RL11^34 0000:4 0101:4

MRL11^34 1000:7:1010101

RL11^34

Indexed RunLists ANDingAnd RL0s 1st, then?

0000101101010101

R11

1RL11 0100:1 0110:2

0RL11 0000:4 0101:1

MRL11 1000:8:01011111

0010100001011111

R34

RL11

1RL34 1011:5

0RL34 0000:2 0101:4

MRL34 0010:3:101 1001:2:10

RL34

0RL11^34 0000:4 0101:4

. . .

RL11^34

Indexed Pure RunLists (no mixed) ANDing. Only need 0RLs!Of course, you need 1RL’s to use as 0RL-comps (maintain 1RLs or construct 0RL-comps on the fly?)To get 1-counts, count 0’s and subtract from total.

0000101101010101

R11

0RL11 0000:4 0101:1 1000:1 1010:1 1100:1 1110:1

0010100001011111

R34

0RL34 0000:2 0011:1 0101:4 1010:1

0RL11^34 0000:4 0101:4 1010:1 1100:1 1110:1

0-count = sum of lengths = 11 1-count=16-11 = 5

0RL11’ 0,4,1,1,2,1,1,1,1,1,1,1,1

Zero RunLists ANDing of 34 and 11’ (with pure1 gaps)(0RL11’ is 0RL11 with a prefixed 0).

0000101101010101

R11

0010100001011111

R340RL34 2,1,1,1,4,1,1,5

1111010010101010

R11’

0RL11’^34 2,

1

1

Intra-Run cursors

1,

3

1

1,

4

1

2,

1

1

6,

1

1

7,

1

1

8,

1

1

9,

1

1

1,

1

2

1,

1

3

1,

1

4

1,

1

5

The 1count of the result is Total minus the 0count or 16 – 13 = 3

So, the coding of this AND program seems straight forward following the animationAn intra-run cursor for each operand and a list cursor for each operand and one for the result.

We, of course, need the 1RLs too (e.g., for 0RL of a complement). Next let’s allow the red gaps to bemixed and insist that the gaps in a 0RL and its corresponding 1RL be compatible.

0rl ANDing of 34 and 11’ with selected mixed gaps, differentiated by a prefix bit. We will use colors on the slides, pure gap=1, mixed gap=0

0000101101010101

R11

0010100001011111

R34

0rl34 2,3:101,4,1,1,5

1111010010101010

R11’

1rl34 0,11:00101000010,50rl11 4,12:101101010101

1rl11 0,6:000010,2,8:01010101

0rl34 2,3:101,4,1,1,5

0rl11’ 0,6:111101,2,8:10 1 01010

2 3:100 4567 4:1010

Note we have to flip mixeds

zmrl11’ 0,4,1,1,2,0,8

Take the philosophy that we will follow a pointer to long mixed runs only when necessary. Otherwise we will sequence straight across.

0000101101010101

R11

0010100001011111

R34zmrl34 2,3,4,1,1,5

1111010010101010

R11’

zmrl11’^34 2,

1

1

zmrl34 2,3,4,1,1,5 101

zmrl11 4,1,1,2,0,8 01010101

3100

3

1

4

1

1

5

2

1

6

3

1

501010

4

1

When the 16-bit window moves left (e.g., add 100 to 0rl11).

zmrl11 0,1,5,1,1,2,0,5 01010

0rl11’ 4,1,1,2,1,1,1,1,1,1,1,1

0000101101010101

R11

0rl11’ 0,1,6,1,1,2,1,1,1,1,1

zmrl11 4,1,1,2,0,8 01010101

Network Security Application(Network security through Vertical Structured data)

Network layers do their own partitioning Packets, frames, etc. (usually independent of any intrinsic data structuring – e.g., record structure)

Fragmentation/Reassembly, Segmentation/Reassembly

Data privacy is compromised when the horizontal (stream) message content is eavesdropped upon at the reassembled level (in network

A standard solution is to host-encrypt the horizontal structure so that any network reassembled message is meaningless.

Alt.: Vertically structure (decompose, partition) data (e.g., basic Ptrees). Send one Ptree per packet Send intra-message packets separately

Trick flow classifiers into thinking the multiple packets associated with a particular message are unrelated.

The message is only meaningful after destination demux-ing Note: the only basic Ptree that holds actual information is the high-order bit Ptree. Therefore

encrypt it! It seems like there ought to be a whole range of killer ideas associated with the concept of using vertical

structuring data within network transmission units Active networking? (AND basic Ptrees (or just certain levels of) at active net nodes?)