Gravi++: Interactive Information Visualization ofHighly Structured Temporal Data

Klaus Hinum1, Silvia Miksch1, Wolfgang Aigner1, Susanne Ohmann2, Christian Popow2, Margit Pohl3, and Markus Rester3

1Institute of Software Technology & Interactive Systems, Vienna University of Technology2Department of Child and Adolescent Neuropsychiatry, Medical University of Vienna3Design and Assessment of Technologies Institute, Vienna University of Technology

Vienna, Austria, http://ieg.ifs.tuwien.ac.at/projects/in2vis/

AbstractTracking and comparing psychotherapeutic dataderived from questionnaires involves a largenumber of highly structured, time-oriented pa-rameters. Descriptive and other statistical meth-ods are only suited for partial analysis. There-fore, we invented a spring-based interactive In-formation Visualization method for analysingthese data more in-depth. With our method theuser is able to find new predictors for a positiveor negative course of the therapy due to the com-bination of various visualization and interactionmethods.

1 IntroductionVisualization tools have been used in the medical domainfor a long time. The majority of methods and tools wasdeveloped for the field of scientific visualization, for exam-ple 3D volume visualization tasks or computer tomographyvisualizations. In the field of medical treatment planningdifferent kinds of data need to be explored in the reasoningprocess, such as time-oriented patient data or the course ofthe patient state. Information Visualization (InfoVis) tech-niques can be used to support this exploration process andpromote a deeper level of understanding of the data and in-formation under investigation. To complement them, task-specific interaction methods have to be developed.

We are aiming for supporting a psychotherapeutic studyon anorexic girls where high dimensional, abstract, time-oriented medical data is collected. The analysis of thesedata is a challenging process. One way to deal with thisproblem would be to apply statistical methods. However,these methods are limited to prove known hypotheses andcan hardly capture the complex process of therapeutic inter-ventions with not yet discovered hypotheses. To overcomethis limitation, we are investigating interactive informationexploration techniques.

We have developed a new interactive InfoVis method,called Gravi++1, to observe new interdependencies be-tween various kinds of parameters. Gravi++ uses the capa-bilities of the human perceptual system by displaying mov-ing icons on the screen following a spring-based model to

1The name Gravi++ is a mixture of two metaphors. “Gravi”stands for gravitation and the two summation signs stand for twomagnetic plus poles.

facilitate the reasoning process. Furthermore, this processis strongly supported by task-specific interaction methods.

2 The Medical ProblemGravi++ is intended to analyze questionnaires of girls witheating disorders (Anorexia Nervosa (AN)). At the Depart-ment of Child and Adolescent Neuropsychiatry, MedicalUniversity of Vienna, a study is taking place, in which al-ternative therapeutic processes (cognitive behavioral ther-apy) of anorexic girls are compared. The psychologistsand physicians who are working with the girls need to ex-plore the data in an experimental way in order to derivetime-oriented quantitative and qualitative information onthe states of the patients. The important features of theinvolved data structures are their data types, complexity(highly structured), and temporal dimensions.

Data CharacteristicsEvery patient, their parents, and their therapists have toanswer an extensive set of questionnaires before, during(three-monthly), and after the therapy (each time 20 ques-tionnaires per patient, four per parent and one per thera-pist). Each questionnaire consists of about 40 questions.An additional challenge is that the questionnaires are notall asked in the same interval. Some are even used onlyonce. After a normalization process the data derived fromthese questionnaires ranges from ’0’ to ’6’. This range canmean ’yes’ to ’no’, but can also stand for nominal valueslike ’feeling guilty after eating a meal’.

Example questions of different questionnaires are:FAMOS14: ’To relax is’: (1) totally unimportant (2) unim-portant, (3) does not matter, (4) rather important to me, (5)extremely important to me.MRFSF1: ’I treat myself to tranquility and recreation’: (1)no, (2) rather no, (3) rather yes, (4) yes.EAT13: ’I feel sick after eating’: (0) never, (1) seldom, (2)sometimes, (3) often, (4) very often, (5) always.

Some questions are concatenated to so called predictors.These predictors should give an idea about a specific statusof a patient and is used to predict the further developmentof the therapy.

The following predictors have been identified during ouranalysis: (1) Predictors for a negative therapeutic courseare ’lacking close friendships’, ’comorbid personality dis-order’, ’low self directedness’, ’lacking sincerity in answer-ing questionnaires because of highly social desirable an-

person icon

question 3

question 2question 1

spring representinganswer

Figure 1: General Principle: A person is attracted by threequestions. The answer is modelled with an invisible spring.The higher the question is answered, the stronger the springpulls to the black disk. We plotted the springs to illustratethis general principle.

swering style’, and ’denying disease’; (2) Predictors forpositive course are a ’decreasing depression score after halfa year of cognitive behavioral therapy (CBT)’ or an ’in-creasing value of feelings of pleasure by doing favoring ac-tivities after 3 months of CBT’.

The task of finding new predictors, like the ones men-tioned above, lead to the development of the new visualiza-tion technique Gravi++.

3 Related WorkDifferent kinds of InfoVis methods were developed in thelast years. Important features of InfoVis methods are tosupport the exploration process of complex, heterogeneousdata promoting a deeper level of understanding of the dataand information and to foster new insights into the under-lying exploration process and the data themselves[Ware,2000; K. Card, 1999].

Medical highly structured data, such as psychothera-peutic data, impose an additional challenge due to theircomplexity and their temporal dimensions. Worm Plots[Matthews and Roze, 1997], the Zoom Star solution[Noirhomme-Fraiture, 2002], the TimeWheel[Tominskietal., 2003], the Table Lens[Rao and Card, 1994], Stardi-nates[Lanzenbergeret al., 2003b], and LinkVis [Lanzen-bergeret al., 2003a] are techniques that try to visualizeand explore such kinds of data. However, none of thesetechniques provide enough interaction possibilities to findnew interdependencies (such as predictors) and explorethe data thoroughly. Furthermore, we are dealing with ahuge amount of highly structured time-oriented data, whichneeds appropriate methods to analyze and to discover pat-terns. Therefore, we created Gravi++ with the help of thefollowing concepts.

We were inspired by the outstanding idea of the VibeSystem[Hendleyet al., 1995] to position document iconsaccording to the occurrence of keywords with a spring-based system and adapted that idea for our core visual-ization. A similar approach was proposed in the RAD-VIZ method [Brunsdonet al., 1998] to map a set of m-

MRFSF1

FAMOS14

EAT13

Person 1

Person 3

Person 2

Person 4

Figure 2: Simple example of finding a cluster: All personsgave similar answers to the three questions. Therefore, acluster of all person’s icons can be seen.

dimensional points onto two dimensional space. The ideafor the different metaphors was inspired by the work of[McGinn and Picking, 2003]. Animation is one way toeffectively visualize temporal changes, which is shown in[Nakakoji et al., 2001]. Furthermore, to present anotherview of multidimensional data, we used Star Glyphs suchas those presented in the XmdvTool[Ward, 1994].

4 Gravi++4.1 ConceptThe human perceptual system has a remarkable abilityto organize and locate things spatially, judge comparativesizes, distinguish between a large range of colors and pat-terns and perceive motion[Olsenet al., 1993]. Gravi++tries to utilize these human capabilities by positioning iconson the screen. There are two kinds of icons representing (1)patients and (2) questions from the questionnaires respec-tively (compare Figure 3). Every patient is attracted by thequestions according to the answer she gave. This is mod-eled with a spring-based system. Every question is con-nected with every person by a spring. This is illustrated inFigure 1 showing a person who is attracted by three ques-tions.

The strength of the individual springs depends on the an-swer the patient gave. This way, every persons’ icon posi-tion on the screen identifies how she answered the involvedquestions. This leads to the formation of clusters of personswho gave similar answers. Because of the fact that thingsthat are close together are perceived as a group, the find-ing and differentiation of clusters is an easy task for the hu-man perceptual system according to the Gestalt Laws[Kof-fka, 1935]. In Figure 2 all persons gave similar answersto the questions MFRS1, FAMOS14, and EAT13. There-fore, their icons form a cluster near EAT13. Furthermore,this tells us, that all persons have answered EAT13 with ahigher value than the other questions.

The size of a person’s icon can be mapped to any addi-tional parameter (for example to the body mass index of thepatients) or to the attraction force. In the second case thesphere is larger if it is attracted by higher values. This fea-

ture helps discriminating different icons that are attractedby the same values with a different coefficient. For exam-ple, if a person has answered the questions one, two, andthree with answer number one, the icon is on the same placeas a person that has answered each question with answernumber five.

To visualize the changing values over time, Gravi++ usesanimation. The position of each person’s icon change overtime allowing to trace, compare and analyze the changingvalues. Alternatively the change over time can also be rep-resented by traces. The size and path of the person’s icon isshown corresponding to all time steps or only to a restrictedsubset like the previous and the next time step.

To visualize the exact values of each question, ringsaround the question’s icon can be drawn. The ring sizecorresponds with the attraction to the question. To avoidoverlapping rings with the same value, they are put closelyside by side.

In addition, Star Glyphs[Ward, 1994] can be shown,which communicate the exact values. The edges of the StarGlyph are connected with the corresponding question ringsand both are drawn in the same color as the person icon.This helps the user to identify the corresponding person.

Missing data is handled by the system in two ways. Ifa person has answered no questions at a specific time step,the icon of the person is transparent. If a value of the size ofa person’s icon is missing, the icon is shown with a specialmarking.

Gravi++ is intended for a restricted parameter space. Themore questions are selected, the smaller is the influence ofa single question on the position of the person’s icons. Fur-thermore, too many person icons lead to clutter because ofoverlapping icons. To select a suitable subset of parame-ters, we have implemented a list-based overview visualiza-tion.

The main aim of the visualization is to derive predic-tors. In the following, different kinds of interactions areexplained to support that task.

4.2 Interactions with Gravi++Gravi++’s main intention is to provide functionalities to getnew insights into the data, like finding clusters or similari-ties in the movement over time. Furthermore, icons that aredrifting apart can give important clues regarding the data.For this purpose Gravi++ provides a set of interactions.

All interactions can be classified in three categories: (1)interactions on question’s icons, (2) interactions on per-son’s icons, and (3) general interactions. In the followingthese categories are explained in detail.

Interactions on Question’s Icons• Add or remove a question.

• Change the position of a question:The position can bechanged either freely or arranged on a circle. Further-more, the user can choose to evenly space the distancebetween the questions on the circle. When changingthe position of the questions, the corresponding po-sitions of the person’s icons change simultaneously.With this feature the user can interactively changethe positions of the questions to find new clusters orother interesting visual structures. The traces and the

Star Glyph also change their form automatically whendragging a question around. This feature is called ’livepreview’.

• Change the influence of a question(seestrengthj insubsection 4.3): This enables the user to emphasizethe influence of a chosen question.

• Add complementary question:By executing this op-tion on a question, a new question with complemen-tary values is created.

• Highlighting a question:By highlighting a question,only the highlighted question’s rings are shown. If noquestion is highlighted, all rings are shown.

• Hide question icons:All question icons can be hiddento reduce clutter and facilitate the analysis of traces orthe Star Glyph.

Interactions on Person’s Icons• Add or remove a person’s icon.

• Change the parameter representing the sphere size:You can map the parameters of a single question, thepower of attraction, or an increasing size over time tothe sphere size.

• Change the speed of movement:Adjust how fast theperson icons are moving to their destination.

• Hide person icons:All person icons can be hidden toreduce clutter and facilitate the analysis of traces orStar Glyphs.

• Show traces:The user has the option to show tracesranging over all time steps or only a restricted subset.

General Interactions• Save and load current settings and visualization:The

position of all elements and all settings can be savedfor later analysis or logging purpose.

• Show Star Glyph:The user can superimpose a StarGlyph of the currently displayed data set to clarify theactual values. The corners of the Star Glyph corre-spond to the positions of the questions.

• Inverse Star Glyph:The edges of the Star Glyph arepainted either to the center of the visualization or out-wards.

• Next and previous time step function:This changesthe time parameter to the next or previous one.

• Direct selection of time steps:The user can directlychoose the time step on a time line or in a list.

• Highlighting: Persons and questions can be high-lighted.

• Show grid: The user may fade in a grid in the back-ground. This helps judging distances and sizes.

4.3 AlgorithmAs we explained the idea and concept of Gravi++ in sec-tion 4.1, now we present the algorithm for positioningthe person’s icons. The force from each personi andeach questionj for each axis (in our case for the x andy axis) isxforcei =

∑nj=0 strengthj · valueij · f(qx)

Figure 3: Typical application case with Gravi++.

andyforcei =∑n

j=0 strengthj · valueij · f(qy), wherestrengthj stands for the manually set strength multipli-cator of the questionj (can be altered in the user inter-face and ranges from 0 = no attraction ton = an arbitraryvalue). valueij stands for the answer of personi to ques-tion j. f(qx) andf(qy) stands for the attraction over thedistance for each axis. Currently, we use a linear functionthat grows stronger over the distance:f(qx) = qxj − pxi,f(qy) = qyj − pyi. Hereqxj andqyj stand for the x, ycoordinates of the questionj andpxi andpyi stand for thex, y coordinates of the personi.

Now we can analytically solve the equations by substi-tuting the force with zero (xforcei = 0 andyforcei = 0),to find the final position for each person (pxi, pyi).

4.4 ImplementationGravi++ was implemented as a prototype during the in2visproject2. The prototype was implemented in MacromediaFlash MX 2004 because of its rapid visual developmentpossibility.

The system consists of two visualizations that workclosely together. There is an overview visualization(ListVis), to select a subset of a large data set. This subsetcan then be analyzed with the main visualization Gravi++.The data exchange between these two modules is imple-mented by drag and drop. You can simply drag a person orquestion to Gravi++ and explore it further. Both modulessupport linking and brushing.

Gravi++ was implemented with different metaphorswhich can be exchanged. In the first of the currently twoimplemented metaphors the persons are symbolized by ironspheres that are attracted by magnets standing for the ques-tions. The circles around the magnets stand for the mag-

2Interactive Information Visualization: Exploring and Sup-porting Human Reasoning Processes

netic fields and visualize the concrete answers. The secondmetaphor shows the persons as people and maps the ques-tions to paintings in an art exhibition. Here the metaphor isexplained by persons who are attracted by beautiful paint-ings. The person moves closer to those paintings she likes.The circles around the images stand for how much the per-son likes the image or not. Other metaphors can be imple-mented by exchanging the icons representing persons andquestions.

The persons are color-coded with twelve distinct colorsas Colin Ware proposed in[Ware, 2000]. To enable the userto recognize icons that lie on top of each other, all iconsare drawn slightly transparent. To control the time steps, atime line was implemented showing the current time stepand providing the “video-like” controlsplay, rewind, stop,go to first/last frame, go to next/previous frame. Tool tipsgive complete information on the data item pointed at withthe mouse.

In Figure 3 you can see a screenshot of a typical applica-tion case.

5 Example: How to Find a PredictorPredictors allow to assess the development of parametersafter a certain treatment step had been performed. There-fore, we have to analyse the differences in the first answersand compare them with the differences over all timesteps.This can be done by watching the animations over time oranalysing the plotted traces of the persons’s icons. Further-more, the change over all timesteps can also show somesort of pattern that indicates a new predictor.

Experimenting sessions with our partners of the Depart-ment of Child and Adolescent Neuropsychiatry identifiedthe following steps to find predictors. We will illustratethese steps using an example of our dataset consisting ofnine patients and 73 questions in six time steps:

Figure 4: The left-hand screenshot shows four questions and no discernable pattern in the traces. The right-hand screenshotshows the same questions ordered differently with two easyly distinguishable clusters in the traces.

1. We choose the questionnaires ASW (inverted)3, BDI4,MR EVA5, MR SOC6 and all persons from theoverview visualization and add them to Gravi++.

2. With the enabled option “Full Traces” we can analysethe traced paths of the persons over time.

3. By moving around the questions we try to find clus-ters of traces. In our example we have nine differ-ent persons. They are color coded the following way:The icons and traces with dark grey shades had nota favourable outcome in the therapy. The icons andtraces with light grey shades had a positive therapyprogress. This means we have to find clusters ofevenly coloured traces.

4. By positioning MR EVA and MR SOC on the one sideand ASW (inverted) and BDI on the other side wecan build two clusters of light and dark traces. In theleft screenshot on Figure 4 you can see the four ques-tions before positioning (with no identifyable cluster-ing of the traces) and on the right screenshot after-wards (with two identifyable clusters of light and darktraces).

5. By moving through the timesteps we can see that therandomly located persons in the first timestep, com-pose two clusters in the second and all later timesteps.Therefore, we can conclude that high values of BDIand the inverted ASW after three months therapy pre-dict a bad outcome of the therapy, whereas high values

3ASW: The Self-Efficacy Scale Index is a summary of the testitems assessing self-esteem and strategies to cope with difficultsituations independently.

4BDI: The Beck-Depression-Index describes the severity ofdepression.

5MR EVA: The Marburg Inventory Index specifies the feelingof pleasure in hedonistic behavior.

6MR SOC: The Sense of Coherence Scale Index indicates theintensity of comprehensibility, manageability and meaningfulnessas the internal psychological mechanism mediating the effects ofexternal stressors and resources on psychological dysfunction.

of MR EVA and MR SOC predict a good outcome.Furthermore, we verify this claim by watching themovement over all remaining timesteps. In our exam-ple the persons with a positive therapy outcome stillmove further to the pole with MR EVA and the MRSOC and the persons with a negative outcome to theother pole. This means we can conclude that low BDIvalues and high MR EVA, MR SOC, and ASW valuescould be a predictor for a positive therapy outcome.This is confirmed by the clinical impression that pa-tients who repsond more rapidly to the primary ther-apeutic goals (enjoying pleasure and being in a goodmood), have a better outcome.

6 Benefits and Limitations

The visual elements of Gravi++ use several advantagesfrom a cognitive perspective[K. Card, 1999]. Especially,the interactive manipulation can help the user get new in-sights through the data. This can be used to formulate andtest a hypothesis on the data e.g., to find a new predic-tor. The combination of different visualization techniques,like Star Glyph, traces, an overview visualization, and theGravi++ core itself increases the possibilities to find newinsights. What is special about Gravi++ is the combinationof these advantages and its orientation on medical data.

Nevertheless, some problems and shortcomings ofGravi++ are still not solved. Incomplete data leads to in-comparable person icon positions because there is no at-traction from questions that were not answered. A solutionto this problem could be to use the value from the last timestep, to use an average value, or a default value. A restric-tion of Gravi++ is the parameter space. Too many questionslead to clutter and make the interactive change very diffi-cult. The impact of one question on the person’s positiondeclines the more questions are shown. Furthermore, if toomany persons are analyzed, many icons would overlap andit would be difficult to interact with the representation. Thevalues of the rings surrounding each question would be in-

distinguishable. This happens because rings representingthe same value are not shown on top of each other but nextto each other. The solution for such situations ist to usehighlighting of subgroups for detailed exploration.

7 Conclusion and Future WorkWe have presented an interactive InfoVis method, calledGravi++, which addresses the particular features of ab-stract, highly structured data which are acquired duringcognitive behavioral treatment (CBT) of anorexia ner-vosa in adolescent girls. This data is difficult to exploreby descriptive and other statistical methods. Because ofGravi++’s various visualization and interaction techniquesit is an appropriate method for finding new predictors inthe data. Our cooperating psychologists see various appli-cation areas for this InfoVis technique within their clinicalstudy of anorexic girls.

A possible extension to Gravi++ could be an algorithmthat automatically positions the questions to find clusters ofpersons.

In the next step we plan to integrate the results of an in-depth user interface study with about 20 participants. Af-terwards, a large number of subjects will be involved in astudy, that will compare Gravi++ with supervised machinelearning and exploratory data analysis to get new insightson the impact on the human reasoning process.

AcknowledgementsThis project is supported by The Vienna Science and Tech-nology Fund (’Wiener Wissenschafts-, Forschungs- undTechnologiefonds’ - WWTF), grant WWTF CI038. Manythanks to the anonymous reviewers who helped us to im-prove our paper.

References[Brunsdonet al., 1998] C. Brunsdon, A. S. Fotheringham,

and M. E. Charlton. An Investigation of Methods forVisualising Highly Multivariate Datasets.Case Studiesof Visualization in the Social Sciences. Joint InformationSystems Committee / ESRC, pages 55–80, 1998.

[Hendleyet al., 1995] R. J. Hendley, N. S. Drew, A. M.Wood, and R. Beale. Narcissus: Visualising Informa-tion. In Nahum D. Gershon and Steve Eick, editors,Proceedings of the IEEE Symp. Information Visualiza-tion (InfoVis ’95), pages 90–96. IEEE Computer SocietyPress, 30–31 1995.

[K. Card, 1999] B. Shneiderman K. Card, J. D. Mackinlay.Readings in Information Visualization. Morgan Kauf-man, 1999. Chapter 1.

[Koffka, 1935] Kurt Koffka. Principles of Gestalt Psy-chology. Harcourt-Brace, 1935.

[Lanzenbergeret al., 2003a] Monika Lanzenberger, SilviaMiksch, Susanne Ohmann, and Christian Popow. Ap-plying Information Visualization Techniques to Captureand Explore the Course of Cognitive Behavioral Ther-apy. InProceedings of the 2003 ACM symposium on Ap-plied computing (SAC ’03), pages 268–274. ACM Press,2003.

[Lanzenbergeret al., 2003b] Monika Lanzenberger, SilviaMiksch, and Margit Pohl. The Stardinates - Visualiz-ing Highly Structured Data. InProceedings of the Sev-enth International Conference on Information Visualiza-tion (InfoVis ’03), pages 47–52. IEEE Computer Soci-ety, 2003.

[Matthews and Roze, 1997] Geoffrey Matthews and MikeRoze. Worm Plots.Computer Graphics and Applica-tions, IEEE, 17(6):17–20, 1997.

[McGinn and Picking, 2003] John McGinn and RichardPicking. The Argument-as Metaphor in Decision-making Visualisation. volume Seventh InternationalConference on Information Visualization (InfoVis ’03),pages 596–599. IEEE Computer Society, 2003.

[Nakakojiet al., 2001] Kumiyo Nakakoji, AkioTakashima, and Ysuhiro Yamamoto. CognitiveEffects of Animated Visualization in Exploratory VisualData Analysis. Fifth International Conference onInformation Visualisation, IEEE Computing Society(InfoVis ’01), pages 77–84, 2001.

[Noirhomme-Fraiture, 2002] M. Noirhomme-Fraiture. Vi-sualization of Large Data Sets: The Zoom Star Solution.International Electronic Journal of Symbolic Data Anal-ysis, 0(0), 2002.

[Olsenet al., 1993] Kai A. Olsen, Robert R. Korfhage,Kenneth M. Sochats, Michael B. Spring, and James G.Williams. Visualization of a Document Collection:TheVIBE System.Information Processing & Management,29(1):69–81, 1993.

[Rao and Card, 1994] Ramana Rao and Stuart K. Card.The Table Lens: Merging Graphical and Symbolic Rep-resentations in an Interactive Focus+Context Visualiza-tion for Tabular Information. InProceedings of the ACMConference Human Factors in Computing Systems (CHI’94), pages 318–322. ACM, 1994.

[Tominskiet al., 2003] Christian Tominski, James Abello,and Heidrun Schumann. Interactive Poster: Axes-BasedVisualizations for Time Series Data. InPoster Com-pendium of IEEE Symposium on Information Visualiza-tion (InfoVis ’03), pages 68–69. IEEE, 2003.

[Ward, 1994] Matthew O. Ward. XmdvTool: IntegratingMultiple Methods for Visualizing Multivariate Data. InProceedings of the conference on Visualization (Vis ’94),pages 326–333. IEEE Computer Society Press, 1994.

[Ware, 2000] Colin Ware.Information Visualization. Mor-gan Kaufmann Publishers, San Francisco, 2000.