KNOWLEDGE ON DEMAND: Knowledge and Expert Discovery Dr. Mark T. Maybury Executive Director...

KNOWLEDGE ON DEMAND:Knowledge and

Expert Discovery

Dr. Mark T. MayburyExecutive Director

Information Technology Division

Knowledge Management Conference Baden Baden, Germany

15 March 2001Organization: G060Project: 05AAV061-C1 MITREhttp://www.mitre.org/resources/centers/it

204/18/23 20:08

MITRE

Knowledge on Demand

•Knowledge Management Strategy

•Knowledge Extraction and Discovery- TIDES, GeoNODE, QANDA, SIAM

•Expert and Expert Community Discovery- ExpertFinder, XperNET

•Facilitating Group Knowledge Creation- KEAN, OWL, SCOUT

•Facilitating Knowledge Communication/Exchange- CVW, TrIM

•Conclusion- Knowledge Management Lessons Learned- Grand Challenges

304/18/23 20:08

MITRE

Why KM?Change is Accelerating

0

2

4

6

8

10

12

14

16

18

0 6 12 18 24 30 36

OpticalNetwork Speeddoubles every 8monthsStoragecapacitydoubles every12 monthsComputingpower doublesevery 18months

dot COM storage requirements double every 90 days

404/18/23 20:08

MITRE

What is KM? An Enterprise Perspective

The strategies, processes, and technologies employed to enable anenterprise to acquire, create, share, and make actionable theknowledge needed to achieve mission objectives

KM Process

Influences

EnablingTechnologiesand Processes

CoPsLeadership

StrategyReward &Recognition

Best PracticeDBs

504/18/23 20:08

MITRE

Framework

604/18/23 20:08

MITRE

KM Enablers

Str

ateg

y

Mea

sure

men

tP

olic

y

Con

tent

Pro

cess

Tech

nolo

gyC

ultu

re

State 2Harvesting the Benefits

KM Targets

Enterprise Processes

Knowledge Discovery

Tool/Process integration

Knowledge Creation and Re-use Impact

State 1Fostering Knowledge Development

Common KM Understanding

Center Pilots

Consolidated Resource View

Greater Tool Standardization

Knowledge-Sharing

Knowledge-Enabled Outcome States

State 0Where We Were

Ad Hoc Processes

Local Initiatives

Disparate Views of Resources

Low Tool Standardization

Collaboration Valued

State “V”Ultimate

Vision

Embedded KM

Known Knowledge Value

Pervasive Infrastructure

Innovative Outcomes

804/18/23 20:08

MITRE

APQC Model of Stages of KM Implementation

Develop interest and enthusiasm No formal businesscase; belief in the

valueDefine KM in termspeople understand

Capitalize on intranet Understand organizational readiness

Select pilots oridentify grassroot efforts

Businessobjectives arespecific to pilots

Form a cross-functional KMtask force

Scale up; buildcapability

Business caseand measuresbecome moreformal

KM coordinationteam

Identify roles and resources for the KM function Establish awards and recognition

KM embedded inbusiness model

Organizationalalignment

Project work withactivity andknowledge basesupport

Standards

Pilot Path

StrategicPilots

OpportunisticPilots

Improve

Expand

Disengage

Way ofdoingbusiness

Decision

Support pilots Business case ispotential gainfrom pilots

Share pilotlessons learnedDevelop methodologies

that can be replicated

KM Portfolios of KM Best Practice Companies (APQC, 2000)

World Bank

Seimens

HP Consulting

MITRE

Chevron

Resource Communications Collaboration Work Application

Resource Tool (Pull) - Yellow Pages, Best Practice DBs, Search Engines

Communications - e-mail, Web pages

Collaboration - Access to knowledgeable human resources

Work Application - Project Management, Problem Solutions,Customer Service

X X X

X X X X

X X X X

X X X X

X X X X

Xerox X X X X

Elements Central to KM Approach: Intranet, CoP/Networks, Best Practice Publication

1004/18/23 20:08

MITRE

KNOWLEDGEINFORMATION

INFRASTRUCTURE(KII)

Process

Expertise & Knowledge Discovery

Knowledge Creation

KnowledgeRequirement

Customer(s)

Knowledge TeamFormation

Knowledge Delivery

1104/18/23 20:08

MITRE

Knowledge on Demand







1204/18/23 20:08

MITRE

Today

Documents, Not answers

Documents, Not answers

Multilingual,Multimedia,Multiparty Resources

Multilingual,Multimedia,Multiparty Resources

Vision: Ask Questions, Get Answers

Tomorrow

Question: Where are the leaders of the ELN?

Question: Where are the leaders of the ELN?

Answer: Francisco Galan and Felipe Torres are in the penitentiary at Itagui, Columbia

Answer: Francisco Galan and Felipe Torres are in the penitentiary at Itagui, Columbia

Answers & Drill down

Answers & Drill down

Knowledge Discovery Tools

Collect

Extract(Alembic)

Sources

Summarize(WebSumm)

Cluster/Mine(QueryFlocks)

Collaborate(KEAN, Scout,

ExpertFinder, XperNET)

Translate(CyberTrans)

Browse/Visualize(GeoNODE)

Monitor(SIAM)

Finance Energy Trans. Telecomm Z-Ave

Disseminate/Retrieve(TIDES, QANDA)

Detect, Translate, Extract, Summarize

Tamil document

•Liberation Tigers of Tamil Eelam (LTTE)•Sri Lanka•Velupillai Pirapaharan•Rebellion

Topic Detection

Source: Ron Larson (DARPA TIDES)

The objective of the Sinhala chauvinists was to utilize maximum man power and fire power to destroy the military capability of the LTTE and to bring an end to the Tamil freedom movement. Before the launching of the operation "Jayasikuru" the Sri Lankan political and military high command miscalculated the military strength and determination of the LTTE.

The objective of the Sinhala chauvinists was to utilize maximum man power and fire power to destroy the military capability of the LTTE and to bring an end to the Tamil freedom movement. Before the launching of the operation "Jayasikuru" the Sri Lankan political and military high command miscalculated the military strength and determination of the LTTE.

Summarization

Org Leader HQ LossesSinhala Kumaratunga 3000LTTE Pirapaharan Wanni 1300

Extraction

Today is a significant day in the history of our national liberation struggle, it marks the end of a year during which we have resisted and fought against the biggest ever offensive operation launched by the Sri Lankan armed forces code named "Jayasikuru”...Translation

DARPA TIDES

What is the status of thecurrent Ebola outbreak?

The epidemic is contained;as of 12/22/00, there were 421 cases with 162 deaths

Interaction

CDCWHO

Medicalliterature

Email:ProMed

~ 2500 stories/day

InternlNewsSources

Capture

Translingual

Information Detection Extraction Summarization

Unidentified hemorrhagic fUnidentif ied hemorrhagic f

Ebola hemorrhagic fever in

Re: Ebola hemorrhagi...Re: Ebola hemorrhagi...

ProMEDAnnotatorJane Analyst

10/17/00 19:3710/17/00 20:4210/18/00 7:42

HighNormalNormal

readreplied

ProMED

10/18/00 12:34 High unread

Ebola hemorrhagic fever in

SourceDate

Priority Status

10 99

0 105

1 57

0 10

2 34

0 50

1 1

0 25

5 200

0 45

0 0

0 0

0 0

0 0

0 6

0 32

0 3

0 1

HighNormalHighHigh

Ebola hemorrhagic fever - Uganda

Unfiltered

Outbreak

Cholera

Dengue Fever

Ebola

Infrastructure…

Natural Disas...

Spills

Accidents

WMD Trackin...

Suspicious Il ln...

Suspicious De...

Possible Biolo...

Pathogen threa…

----------------------------

Workspace

Ebola

Drafts

Reports

Disease

Re: Ebola hemorrhagi...

Location

UNKUNKEbolaEbolaEbolaEbolaRabiesRabies

UgandaUgandaUgandaKenyaUganda

IHTProMEDWHO

Joe Analyst

Date

10/14/00 23:0610/15/00 10:5010/16/00 21:4510/17/00 19:12

readreadreadread

unread

Date: 10/16/00

Disease: EbolaDescriptor: hemorrhagic fever

Location: Uganda

Disease Date: 10/14/00Hospital: missionary hospital in Gulu

New cases: at least 7Total cases: 51

Total dead: 31

Ebola hemorrhagic fever -

Ugandan Ministry identifies Ebola virus as the cause of the outbreak. KAMPALA:The dreaded Ebola virus that struck over 300 people in Kikwit, in the DemocraticRepublic of Congo in 1995, has ki lled 31 people in northern Uganda. A UgandanMinistry of Health statement said laboratory tests had revealed that the Ebola viruswas the cause of the epidemic hemorrhagic fever which has been raging in the Guludistrict since September. Three of the dead were student nurses , who treated the firstEbola patients admitted to a Lacor missionary hospital in Gulu town. A task forceheaded by Gulu district administrator, Walter Ochora, has been set up to co-ordinateefforts to control the epidemic. Field officials in Gulu told the Kampala-based New

Http://tides2000.mitre.org/ProMED/10162000/34n390h.html

Uganda

News Repository

CATALYSTCATALYSTEvent ExtractionTime TaggingTDTTranslationSummarization AlertingChange detectionCross-language IR

1604/18/23 20:08

MITRE

TIDES Portal

Metaqueriessupported formultiple sources

Translingual system supportsforeign-languagesources

Multiple mediaexploited

Government &private sourcesutilized

16

Translingual

Information Detection Extraction Summarization

1704/18/23 20:08

MITRE

Geospatial News on Demand (GeoNode)

InformationExtraction

Data MiningAnd Clustering

ttopic

t

Map overview

Topic Timeline

News histogram

•Navigate•Filter•Indexed access•Animate reporting trends•Create reports/ web

BNN Story skim

IndexingAnd NewsModeling

DataAcquisition/Pre-process

GeoNODE Database

News Sources:

Broadcast

Specialist Archives

World Wide Web

Intel. Msg Traffic

1804/18/23 20:08

MITRE

Person1 Person2 Location Support

Mobutu Sese Seko Laurent Kabila Kinshasa 7

0 Two-way associations: people and locations

0 Three-way associations, only one association with support of at least 5 (confidence is 50%: 1/2 of the stories mentioning any item

also mention the other two)

6612 stories, with 13,737 distinct concepts mentioned. The associations between pairs of concepts are ranked by support: the number of documents containing the

pair). All correlations have at least 50% confidence: At least 50% of the stories mentioning one item in a pair also mention the other

Type Value Type Value SupportPerson Natalie Allen Person Linden Soles 117Person Leon Harris Person Joie Chen 53Person Ron Goldman Person Nicole Brown Simpson 19Person Dole Person Bob Dole 18Person Forbes Person Dole 16Person Forbes Organization New Hampshire 15Person Bill Cosby Location Cosby 14Person Pat Buchanan Person Forbes 12Person Steve Forbes Organization New Hampshire 12Person Mobutu Sese Seko Location Kinshasa 10

Data Mining: “Find Significant Warlords in a Region”

1904/18/23 20:08

MITRE

The Web Has Gone Multilingual

•Last year, the web became more than 50% non-English

•Only 15% of Europe's half a billion population speaks English as a first language

•Only 28% speaks English at all

•Only 32% of Web surfers on the European continent consult the Web in English. [Source: Global

Reach:www.euromktg.com/eng/GR/]

•45% of Internet users from non English-speaking countries

•By 2002, analysts estimate that 66% of Internet use and 40% of e-commerce revenue will come from outside the U.S. [Source: IDC]

•300,000 Japanese patents filed annually

"If I'm selling to you, I speak your language. If I'm buying,

dann müssen Sie Deutsch sprechen”

Willy Brandt, former German chancellor

2004/18/23 20:08

MITRE

Open Source Analysis of Latin America

Event Timeline

•10Mar98 Pinochet resigns•17Mar98 Cuban defector, pitcher Orlando

Hernandez•15Apr98 Execution of Paraguayan Angel Breard,

convicted killer in US•21Apr98 Plane crash: Bogota, Columbia to

France•17Oct98 - 27Oct98 Pinochet’s arrest by Scotland

yard while getting medical treatment•26Oct98 House of Lords deny Pinochet

diplomatic immunity•28Nov98 Columbian man surrenders near

Bogota, accused of shooting DEAAgent Moreno

•04Dec98 Iranian Woman has been charged by Argentina's Supreme Court for the1992 bombing of Israeli embassy in Buenos Aires, Argentina

•07Jan99 - 18Jan99 Brazilian Financial crisis

2104/18/23 20:08

MITRE

Multiple Knowledge Sources and

Multiple Applications

GeoNODE BNN

Doc collection

orthe Web

Semi-structured database

RelationalDatabase

Knowledgesources

Userapplications

Q and AEngine

IRS hotlineoperator

MII

Question and Answering - QANDA

Concept + Information Volume + Time + Location + Source => Social Interest

Concept mapped to query sets (large numbers of queries) to address topic specificity and coverage. Query Class, Retrieval Rank and/or Relevance can be used to weight items

Search agent collects from

multiple search engines,

link traversal, and sampling strategies

used to collect relevant items,

and scale up results

to population levels.

Normalized Volume-->Importance

Windowing

collection to

specific time

periods e.g.,

4 quarters per

year

Spatial tags for tracking

by country or region

Source weighting (Optional) for weighting importance based on source type and website structure, e.g., logical location within a Gov. site

Social Interest of Topic X Country Y = F(relevance,scaled- volume,location, time, source)

Issue

Social Indicators Analysis Method (SIAM)

Gartner Group: “Year 2000 World Status,

2Q99: The Final Countdown

Other Gartner Reports (periodic

assessments, Gartner

Interactive Reports,…)

Lou Marcoccio Interview

U.S. Senate : Investigating The Year 2000

Problem: The 100 Day Report”

Department of State: Biannual Consular

Advisory

MITRE Y2K Assessment

US Agency for International Development

(USAID)

International Y2K Cooperation Center

(IY2KCC)

International Monitoring (IM) … a London-

based consultancy

Y2K Assessments are Generally Survey-

Based

Example: Gartner Group

Survey: 330 Questions

Performed Quarterly

600 People Involved

1500+ Companies

3 Calls/Company

11 Universities Provide Analysis

COMPARE Statistic Computed from Survey

Example: Department of State has an

ongoing Survey among its 260 Posts.

Reporting Bias Observed by Gartner...others.

SIAM Processing Costs

New Domain: 1 Day

Quarterly Analysis: 2 Days

Download Time: 1-2 Days for Current Data

Results Analysis and Fine tuning: 2 Days

Today’s Manual Approach

CNTRY Finance Energy Trans. Telecomm Z-Aveargentina 1.476278 2.434932 0.8366 1.9553 1.67579bermuda 0.888375 -0.06594 1.9661 0.1754 0.740966bolivia -0.65204 -0.55287 -0.6155 -0.6380 -0.61458brazil 1.869995 0.27763 0.7220 1.5664 1.109025chile 1.242407 0.065475 1.9946 0.8680 1.042629colombia -0.5338 -0.38638 -0.4981 -0.5096 -0.48196costa_rica -0.39694 -0.29373 -0.4429 -0.2979 -0.35789ecuador -0.6348 -0.54902 -0.6117 -0.6289 -0.60608el_salvador -0.6304 -0.52669 -0.5945 -0.6204 -0.59299guatemala -0.65019 -0.54851 -0.6130 -0.6366 -0.61208guyana -0.65496 -0.55555 -0.6146 -0.6410 -0.61652honduras -0.65361 -0.55464 -0.6161 -0.6409 -0.61632mexico 1.656809 2.306324 1.1496 2.0101 1.780716nicaragua -0.60775 -0.52549 -0.5961 -0.6028 -0.58303panama -0.64749 -0.54845 -0.6090 -0.6352 -0.61004paraguay -0.64199 -0.54268 -0.6052 -0.6190 -0.60222peru -0.42739 -0.31945 -0.4733 -0.3972 -0.40431puerto_rico -0.65496 -0.55555 -0.6162 -0.6401 -0.6167uruguay 0.987168 1.667379 1.2754 1.2915 1.305344venezuela -0.33471 -0.2268 -0.4382 -0.3592 -0.33974

High Risk Inferred

Moderate Risk Inferred

Low Risk Inferred

SIAM “Status Board”

VENEZUELACOLOMBIA

BRAZIL

PARAGUAY

URUGUAYARGENTINA

CHILE

BOLIVIAPERU

ECUADOR

MEXICO

NICARAGUA

HONDURAS

COSTA RICA

PANAMA

EL SALVADOR

GUATEMALABELIZE

2504/18/23 20:08

MITRE

Senate Report: “Peru is, across many sectors, either

not Y2k ready or that public information is inadequate.”

… “The Gartner Group and the World Bank offer

contradictory information, ranking Peru as one of the

better prepared in South America.”

mexico 1.656809 2.306324 1.1496 2.0101nicaragua -0.60775 -0.52549 -0.5961 -0.6028panama -0.64749 -0.54845 -0.6090 -0.6352paraguay -0.64199 -0.54268 -0.6052 -0.6190peru -0.42739 -0.31945 -0.4733 -0.3972

SIAM Indicator Board

SIAM Scores Peru as Moderate Readiness Ranking 10th out of 20 Countries

Variance of Opinion

2604/18/23 20:08

MITRE

Knowledge on Demand







2704/18/23 20:08

MITRE

Expert Discovery

•Find global Experts- quick- accurate- comprehensive

•Challenge: Overcome limitations of manually managed skills/expertise databases (e.g. Dataware - experts self nominate)- incomplete - expensive- out of date

2804/18/23 20:08

MITRE

Related Work

•Autonomy- document based (docs, Notes discussions, email)- dynamic expert profiling- Problem: reading/writing not always correlated

w/expertise

•Abuzz- Beehive email routes questions to experts based on

expert profile (must seed this)- Expertise validated by community (+/- satisfaction with

answers) updates profiles- Problem: Seeding/Learning curve

•MIT’s ExpertFinder (Vivacqua)- expertise models from software library use

•Tacit - email based keyword profiling

2904/18/23 20:08

MITRE

Uniqueness

•This work:- Implicitly determines expertise from multiple sources

of evidence including intellectual products (e.g., briefings, papers, web pages) and information seeking actions (e.g., web logs)

- Leverages intranet publishing (staff, corporate news letters), corporate directory services, project leadership information

- Exploits recent advances in information extraction (language processing) technology

Expertise Management Architecture

E-dB

Finder

ServiceBroker

MII

WWW

FinderAgencies

ConsultingGroups

Q&A

Services

Resources

Registration

Qualification

Selection

Expert FinderGoal: Place a user within one phone call of an expert

Enterprise EmployeeProject Database

EmployeesRanked by Mentions

Mentions of Employee inCorporate

Communications

IntegratedEmployeeDatabase

User Issues Simple Query

Relevant EmployeePublications

3204/18/23 20:08

MITRE

ExpertFinder Algorithm

Initial Form

Call Search Engines

Parse Results

Gather All URLs

Find Mentioned People

Find Published People

Combine Info

Add Phone Book Info

Weigh Evidence

Display Results

CachedResults?

yes

no

3304/18/23 20:08

MITRE

Evaluation

•Compare performance of ExpertFinder with (20) expert human resource managers

•Task: Find top 5 corporate experts in a given domain

•Measures- Agreement among humans- Agreement of machine with human(s)

•Precision

•Recall

•Chance: # experts/4500 employees = often less than .1%

3404/18/23 20:08

MITRE

The Questionnaire

1. Who are the top 5 "data mining" experts at MITRE (List them in rank order, most expert first. List as many as you can but no more than 5)?

2. the top 5 "collaboration" experts? 3. the top 5 "chemical" experts? 4. the top 5 "human computer interaction" experts? 5. the top 5 "network security" experts? 6. What is your top area of expertise (in a few words)

and who do you consider to be the top 5 people in the company in your area of expertise?

I am performing an experiment. Your participation will remain anonymous if you so desire and should only take a few short minutes. Please answer the following questions (preferably without any assistance, but if you use assistance indicate what kind you used):

3504/18/23 20:08

MITRE

Human vs ExpertFinder

•Comparison to 20 human resource managers

•Agreement = in top 5 IDed experts

•Precision = # correctly IDed experts / # IDed

•Recall = # correctly IDed experts/ # actual experts

Expert Area Human Agreement(1st, 2nd, 3rd)

ExpFinder Precision

ExpFinderRecall

Data mining 70%, 49%, 24.5% 60% 40%

Chemical 40%, 8%, 0.8% 60% 40%

HCI 90%, 36%, 11% 60% 40%

Network Security 50%, 10%, 0.4% 20% 20%

Collaboration 70%, 35%, 17.5% 5% 5%

AVERAGE 63%, 28%, 11% 41% 29%

3604/18/23 20:08

MITRE

Pattern Analysis•Work/Activity Sampling•Feature Extraction•Topic Detection•Social Network Generation•Community of Practice “Registration”…

Community of Interest Modeling

Organizational Theory

Project InformationWeb Pages

Meetings/Conferences/...Share Folders

Published documents...

Clustering techniquesSocial network analysis methods

Summarization

Emergence…Monitoring

CommunicationSharing

Expert FindingOther Applications

3704/18/23 20:08

MITRE

Expert Communities: XperNet Network

Core Group Expanded Group

Automatic Network Expansion

Network Membership Ratings

0.00010.00020.00030.00040.00050.00060.00070.00080.00090.000

100.000

1 3 5 7 9

11

13

15

17

19

21

23

25

27

29

31

Member Rank

Me

mb

ers

hip

Sco

re

3804/18/23 20:08

MITRE

Extracting Communities of Interest

3904/18/23 20:08

MITRE

Knowledge on Demand







4004/18/23 20:08

MITRE

Words of “Wisdom”

•Bentov’s Law: One’s level of ignorance increases exponentially with accumulated knowledge. When one acquires a bit of new information, there are many new questions that are generated by it, and each new piece of information breeds five-ten new questions. These questions pile up at a much faster rate than does accumulated knowledge. Therefore, the more one knows, the greater his level of ignorance.

•Allen’s Tenet - The strength’s of one’s opinion on any matter or controversy is inversely proportional to the amount of knowledge that person has on that subject.

•BB’s Dictum - In a group, the unknowing will try to teach the lesser-skilled or knowing

4104/18/23 20:08

MITRE

Collaboration Taxonomy

Multiple levels of virtual teamingMultiple levels of virtual teaming

Awareness

Information Sharing

Joint Efforts

Alignment

LeadershipIntent

Incr

easin

g

Inte

ract

ion

Increasing

OverheadCoordination

4204/18/23 20:08

MITRE

Knowledge Exchange and Annotation eNgine (KEAN): Search

SEARCH by

- Subject- Keyword- Employee- Rating Level- Time

boykin

4304/18/23 20:08

MITRE

KEAN: Source Assessment

4404/18/23 20:08

MITRE

Questions answered (with justification) by KEAN (e.g., data mining)

•What information does Chris (expert in data mining) think is useful for data mining?

•What information do people in the data mining community of practice find useful on data mining?

•What information does everyone think is useful on data mining in the past few weeks?

•What information on data mining have I found to be useful in the past?

4504/18/23 20:08

MITRE

KEAN Evaluation

•26 individuals on 295 URLs - length of time on page (“reading” time)- explicit utility rating

•Focused task - directory services questions- Which standards organization defines the X.500

specification?”- “How does LDAP differ from X.500?”- “Name some of the data types that can be stored in an

LDAP attribute.”

•After the experiment, rate utility 1-10 (10 highest)

•Regression test yielded positive correlation - explicit utility = .0113*time read- 66% of all URLs “read” for greater than 78 seconds

were classified as high utility (6-10)

•Time --> utility

4604/18/23 20:08

MITRE

Organizational Wide Learning (OWL):Word Command Usage by Type

Command Usage by Type

0% 10% 20% 30% 40% 50% 60%

Edit

File

Format

View

Window

Insert

Tools

Table

Help

4704/18/23 20:08

MITRE

OWL Data: Word’s Top 10 Commands

Sequence Command PercentCumulativePercent

1 Edit Delete 34.2% 34.2%2 File Save 10.5% 44.8%3 File Open 8.7% 53.5%4 Edit Paste 7.9% 61.4%5 File DocClose 5.1% 66.5%6 Edit Copy 4.2% 70.7%7 Format Bold 3.7% 74.4%8 File Print 2.8% 77.2%9 Edit Cut 2.4% 79.7%

10 File SaveAs 1.7% 81.3%

4804/18/23 20:08

MITRE

Organizational Knowledge & Ignorance Some individuals never use a number of the more frequently-used commands

Count of Users & Usage

0

5

10

15

20

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49

Command Sequence

Use

rs

1

10

100

1000

10000

100000

Usa

ge

Users Usage

4904/18/23 20:08

MITRE

OWL recommends what to learn next(unique to each individual at each point in time)

USER #314 ExpectedObservedInstructionEdit Paste 170 274 OKEdit Delete 129 0 NewEdit Copy 107 97 OKEdit Cut 48 100 OKEdit Undo 16 14 OKEdit Find 12 1 MoreEdit SelectAll 9 12 OKEdit DeleteWord 4 0 NewEdit Replace 3 0 NewEdit PasteSpecial 2 0 New

5004/18/23 20:08

MITRE

Do OWL Users Use Help?

Use of MS Word Help & Office Assistant

0

2

4

6

8

10

12

14

16

Daily Weekly Monthly Annually Never

Frequency of Use

Num

ber

of U

sers

Office Assistant Help

• More than half (12/20) use help at least monthly.

• Only a few (6/20) use Office Assistant

5104/18/23 20:08

MITRE

Cooperative Searching (Monitoring Group Information Seeking Activities)

-- a multi-user collaborative retrieval tool. The “next generation” in IR systems addresses multi-user, coordinated searching, shared analysis, and has a built-in recommender system. Tracks topics, users, and provides a persistent knowledge store.

5204/18/23 20:08

MITRE

Cooperative Searching

Hypothesis

Group (coordinated) searching can be more effective than multiple (independent) searchers

working autonomously

5304/18/23 20:08

MITRE

Cooperative Searching-- Initial Prototype

Local Shared Workspace

Collect

Users Generate Ad Hoc Queries

Off-line Queries or Web-page Monitoring SupportedUsers Generate Ad Hoc Queries

Off-line Queries or Web-page Monitoring Supported

Users Generate Task FoldersUsers Generate Task Folders

5404/18/23 20:08

MITRE


Collect

Retrieved Information is Organized by Domain

Search Engine Statistics Provided

Offline Cluster Analysis and Categorization Provided

Retrieved Information is Organized by Domain

Search Engine Statistics Provided

Offline Cluster Analysis and Categorization Provided


5504/18/23 20:08

MITRE



Collect

User Actions Infer Relevance

Ratings/Annotations/Actions Are Stored User Actions Infer Relevance

Ratings/Annotations/Actions Are Stored

Access Count and Evaluation Status Provided

5604/18/23 20:08

MITRE


Collect

Rated Items with Annotations are

integrated into a Shared Context Rated Items with Annotations are

integrated into a Shared Context


5704/18/23 20:08

MITRE

Knowledge on Demand







5804/18/23 20:08

MITRE

Collaborative Virtual Workplace (cvw.sourceforge.net)

5904/18/23 20:08

MITRE

Translingual Instant Messaging (TrIM)

•Integration of - Simple Instant Messaging Protocol (simp.mitre.org) - CyberTrans machine translation framework

•Supports multilingual chat

•Research Issues:- Quality of conversational translation using document

translation engines- Presentation (monolingual, multilingual)- Data collection to learn language models

6004/18/23 20:08

MITRE

Knowledge on Demand







Knowledge Management Capability Maturity Model (KM CMM)

Level 4: Managed• Integrated knowledge processes• Quantitative process management

Level 4: Managed• Integrated knowledge processes• Quantitative process management

Level 5: Optimizing• Business process alignment• Process change management

Level 5: Optimizing• Business process alignment• Process change management

Wherewe are

‘00

Where we aregoing

‘01

Where we want to

be

Level 1: Initial• Adhoc processes• Partial technical infrastructure

Level 2: Repeatable• Program planning • Content QA process• Requirements process • KFP identification

Level 3: Defined• Organizational processes • Knowledge mapping• Intergroup coordination • Training program

Level 0:Not Practiced• Failure to perform KM• Culture counter to learning, sharing

6204/18/23 20:08

MITRE

Strategy

Peter Senge

“Learning Organizations”

Process

Takeuchi and Nonaka

“Organizational Knowledge Creation”

Benchmarking

Norton and Kaplan

“Balanced Scorecard”

a

C I I SCENTER FOR INTEGRATEDINTELLIGENCE SYSTEMS

MITRE

6304/18/23 20:08

MITRE

Lessons Learned

People, and the cultures that influence their behaviors, are the single most critical resource for successful knowledge creation, dissemination, and application. Understand and influence them.

Cognitive, social, and organizational learning processes are essential to the success of a knowledge management strategy. Focus your strategy on enhancing these processes.

Measurement, benchmarking, and incentives are essential to accelerate the learning process and to drive cultural change. Create a tailored balanced scorecard to target what you want to improve.

Knowledge management programs can yield impressive benefits to individuals and organizations if they are purposeful, concrete, and action-oriented. Make yours so.

In times of profound change, learners inherit the Earth,

while the learned find themselves beautifully

equipped to deal with a world that no longer exists.

- Al Rogers -

6504/18/23 20:08

MITRE

Some Grand Challenges

•User, Group and Organization Modeling, including knowledge, beliefs, goals and plans

•(U, G, O) Tailored presentation of knowledge

•Ontological integration of distributed DB & KB

•Universal knowledge access independent of user physical, perceptual, cognitive, cultural characteristics

•Organizational strategies for knowledge sharing

•Knowledge strategies in global, multicultural enterprises

•Privacy and Security

6604/18/23 20:08

MITRE

Events of Interest

•8th Internat. Conference on User ModelingSonthofen, Germany July 13-17, 2001www.dfki.uni-sb.de/cgi-bin/um2001

•Workshop on Human Language Technology and Knowledge ManagementToulouse, FranceJuly 6-7, 2001www.elsnet.org/acl2001-hlt+km.html

•Our work: www.mitre.org/resources/centers/it

Joint EACL - ACL Meeting

6704/18/23 20:08

MITRE

Acknowledgements

•Knowledge Management - Cynthia Small, Jean Tatlias

•TIDES - Lynette Hirschman, Jay Ponte et al.

•GeoNODE - Rod Holland, John Griffith et al.

•QANDA - Marc Light

•OWL - Frank Linton

•CVW - Jay Carlson, Deb Ercolini et al.

•TrIM - Rod Holland, John Ramsdell, Flo Reeder, Jay Carlson, Justin Richer, Galen Williamson, Michael Krutsch, Keith Crouch, Keith Miller

•SIAM, XperNET, SCOUT - Ray D’Amore, Manu Konchady

•KEAN - Daryl Morey, Tim Frangioso

•ExpertFinder - Dave Mattox, Inderjeet Mani, David House

KNOWLEDGE ON DEMAND:Knowledge and

Expert Discovery

Dr. Mark T. MayburyExecutive Director

Information Technology Division

Knowledge Management Conference Baden Baden, Germany

15 March 2001Organization: G060Project: 05AAV061-C1 MITREhttp://www.mitre.org/resources/centers/it

KNOWLEDGE ON DEMAND: Knowledge and Expert Discovery Dr. Mark T. Maybury Executive Director...

Documents

Transcript of KNOWLEDGE ON DEMAND: Knowledge and Expert Discovery Dr. Mark T. Maybury Executive Director...