Solving the Unstructured Data Problem – Case Study the Unstructured Data Problem – Case Study...
Transcript of Solving the Unstructured Data Problem – Case Study the Unstructured Data Problem – Case Study...
![Page 1: Solving the Unstructured Data Problem – Case Study the Unstructured Data Problem – Case Study Subodh Manicka, ... Federal Institute for Drugs and Medical ... Cognos BI Complaints](https://reader030.fdocuments.in/reader030/viewer/2022021505/5acc140f7f8b9a875a8c2a19/html5/thumbnails/1.jpg)
© 2017 IBM Corporation
Solving the Unstructured Data Problem – Case Study
Subodh Manicka, Exec IT Architect, IBM Corporation
Randall Ho, Senior IT Architect, IBM Corporation
![Page 2: Solving the Unstructured Data Problem – Case Study the Unstructured Data Problem – Case Study Subodh Manicka, ... Federal Institute for Drugs and Medical ... Cognos BI Complaints](https://reader030.fdocuments.in/reader030/viewer/2022021505/5acc140f7f8b9a875a8c2a19/html5/thumbnails/2.jpg)
Function of a Government
• A primary function of government is to protect citizens from harm
� Personal safety and security
� Unfair trade practice
� Etc.
• We expect our government to
� Keep the citizens safe from bad people
� Regulate automobile and consumer product safety
� Regulate drug and medical device safety
� Regulate medical device safety
� Regulate unfair trade practice
1
![Page 3: Solving the Unstructured Data Problem – Case Study the Unstructured Data Problem – Case Study Subodh Manicka, ... Federal Institute for Drugs and Medical ... Cognos BI Complaints](https://reader030.fdocuments.in/reader030/viewer/2022021505/5acc140f7f8b9a875a8c2a19/html5/thumbnails/3.jpg)
How does the Government address this?
• Through Laws and Regulations
• Investigating safety issues or concerns reported by citizens
• Defending the country from enemies of the state
2
![Page 4: Solving the Unstructured Data Problem – Case Study the Unstructured Data Problem – Case Study Subodh Manicka, ... Federal Institute for Drugs and Medical ... Cognos BI Complaints](https://reader030.fdocuments.in/reader030/viewer/2022021505/5acc140f7f8b9a875a8c2a19/html5/thumbnails/4.jpg)
3
Headlines Related to Automobile Safety
![Page 5: Solving the Unstructured Data Problem – Case Study the Unstructured Data Problem – Case Study Subodh Manicka, ... Federal Institute for Drugs and Medical ... Cognos BI Complaints](https://reader030.fdocuments.in/reader030/viewer/2022021505/5acc140f7f8b9a875a8c2a19/html5/thumbnails/5.jpg)
4
Headlines Related to Consumer Product Safety
![Page 6: Solving the Unstructured Data Problem – Case Study the Unstructured Data Problem – Case Study Subodh Manicka, ... Federal Institute for Drugs and Medical ... Cognos BI Complaints](https://reader030.fdocuments.in/reader030/viewer/2022021505/5acc140f7f8b9a875a8c2a19/html5/thumbnails/6.jpg)
5
Headlines Related to Citizens Comments to Regulations
![Page 7: Solving the Unstructured Data Problem – Case Study the Unstructured Data Problem – Case Study Subodh Manicka, ... Federal Institute for Drugs and Medical ... Cognos BI Complaints](https://reader030.fdocuments.in/reader030/viewer/2022021505/5acc140f7f8b9a875a8c2a19/html5/thumbnails/7.jpg)
6
Impact to Business
![Page 8: Solving the Unstructured Data Problem – Case Study the Unstructured Data Problem – Case Study Subodh Manicka, ... Federal Institute for Drugs and Medical ... Cognos BI Complaints](https://reader030.fdocuments.in/reader030/viewer/2022021505/5acc140f7f8b9a875a8c2a19/html5/thumbnails/8.jpg)
Examples of Government Agencies• USA
� Federal Communications Commission
� Consumer Product Safety Commission
� National Highway Traffic Safety Agency
� Food and Drug Administration
• Canada
� Canadian Radio-television and Telecommunications Commission
� Office of Consumer Affairs
� Transport Canada’s Motor Vehicle Safety Directorate
� Health Canada
• Australia
� Australian Communications and Media Authority
� Australian Competition and Consumer Commission
� Vehicle Safety Standards Branch, Department of Infrastructure and Regional Development
� ACPM, ARGMD, Therapeutic goods Administration
7
• UK
� Office of Communication
� Office of Fair Trading
� Driver and Vehicle Standards Agency
� Medicines & Healthcare products Regulatory Agency
• Germany
� Federal Network Agency (Bundesnetzagentur)
� Verbraucherschutzminister
� Federal Ministry of Transport, Building and Urban Development
� Federal Institute for Drugs and Medical Devices
• Austria
� Austrian Regulatory Authority for Broadcasting and Telecommunications
� Federal Ministry of Labour, Social Affairs and Consumer Protection
� Austrian Ministry for Transport, Innovation and Technology
![Page 9: Solving the Unstructured Data Problem – Case Study the Unstructured Data Problem – Case Study Subodh Manicka, ... Federal Institute for Drugs and Medical ... Cognos BI Complaints](https://reader030.fdocuments.in/reader030/viewer/2022021505/5acc140f7f8b9a875a8c2a19/html5/thumbnails/9.jpg)
Process and Action
• Citizens discover safety issue or other concerns
• Citizens voice their concern by submitting complaints
• Citizens provide feedback on upcoming regulations
• To fulfil its mission, government agencies analyze data that they receive and from various sources including citizens and manufactures
• Following investigation, government agencies act on the concern through product recall or change in regulations
8
![Page 10: Solving the Unstructured Data Problem – Case Study the Unstructured Data Problem – Case Study Subodh Manicka, ... Federal Institute for Drugs and Medical ... Cognos BI Complaints](https://reader030.fdocuments.in/reader030/viewer/2022021505/5acc140f7f8b9a875a8c2a19/html5/thumbnails/10.jpg)
Sample Forms
US Automobile – Safecar.gov
9
![Page 11: Solving the Unstructured Data Problem – Case Study the Unstructured Data Problem – Case Study Subodh Manicka, ... Federal Institute for Drugs and Medical ... Cognos BI Complaints](https://reader030.fdocuments.in/reader030/viewer/2022021505/5acc140f7f8b9a875a8c2a19/html5/thumbnails/11.jpg)
Sample Forms
UK Automobile Report Form
10
![Page 12: Solving the Unstructured Data Problem – Case Study the Unstructured Data Problem – Case Study Subodh Manicka, ... Federal Institute for Drugs and Medical ... Cognos BI Complaints](https://reader030.fdocuments.in/reader030/viewer/2022021505/5acc140f7f8b9a875a8c2a19/html5/thumbnails/12.jpg)
Sample Forms
Australia Automobile
11
![Page 13: Solving the Unstructured Data Problem – Case Study the Unstructured Data Problem – Case Study Subodh Manicka, ... Federal Institute for Drugs and Medical ... Cognos BI Complaints](https://reader030.fdocuments.in/reader030/viewer/2022021505/5acc140f7f8b9a875a8c2a19/html5/thumbnails/13.jpg)
Sample Forms
US Consumer Product Safety
12
![Page 14: Solving the Unstructured Data Problem – Case Study the Unstructured Data Problem – Case Study Subodh Manicka, ... Federal Institute for Drugs and Medical ... Cognos BI Complaints](https://reader030.fdocuments.in/reader030/viewer/2022021505/5acc140f7f8b9a875a8c2a19/html5/thumbnails/14.jpg)
Sample Forms
Australian Consumer Product Safety
13
![Page 15: Solving the Unstructured Data Problem – Case Study the Unstructured Data Problem – Case Study Subodh Manicka, ... Federal Institute for Drugs and Medical ... Cognos BI Complaints](https://reader030.fdocuments.in/reader030/viewer/2022021505/5acc140f7f8b9a875a8c2a19/html5/thumbnails/15.jpg)
Sample Forms
14
![Page 16: Solving the Unstructured Data Problem – Case Study the Unstructured Data Problem – Case Study Subodh Manicka, ... Federal Institute for Drugs and Medical ... Cognos BI Complaints](https://reader030.fdocuments.in/reader030/viewer/2022021505/5acc140f7f8b9a875a8c2a19/html5/thumbnails/16.jpg)
Sample Forms
15
![Page 17: Solving the Unstructured Data Problem – Case Study the Unstructured Data Problem – Case Study Subodh Manicka, ... Federal Institute for Drugs and Medical ... Cognos BI Complaints](https://reader030.fdocuments.in/reader030/viewer/2022021505/5acc140f7f8b9a875a8c2a19/html5/thumbnails/17.jpg)
Challenges with the Data
16
• Data volume fluctuates with significant spikes at times
• Citizens misspelling words
• Information is typically a mix of structured data and unstructured text
• Meaning of words may depend on context
� Examples:
• Lead – the element lead in paint that is poisonous or the government regulation that leads to higher consumer cost.
• “Engine caught fire” vs. “Fire department suspects transmission fuel leak”
• Fruit flies like banana
• “Shot” a bear – with a camera vs with a gun
![Page 18: Solving the Unstructured Data Problem – Case Study the Unstructured Data Problem – Case Study Subodh Manicka, ... Federal Institute for Drugs and Medical ... Cognos BI Complaints](https://reader030.fdocuments.in/reader030/viewer/2022021505/5acc140f7f8b9a875a8c2a19/html5/thumbnails/18.jpg)
The Architecture
![Page 19: Solving the Unstructured Data Problem – Case Study the Unstructured Data Problem – Case Study Subodh Manicka, ... Federal Institute for Drugs and Medical ... Cognos BI Complaints](https://reader030.fdocuments.in/reader030/viewer/2022021505/5acc140f7f8b9a875a8c2a19/html5/thumbnails/19.jpg)
18
IT Context Diagram
Cognos BI
Complaints / Issues
Unstructured Data
Analytics
Data Warehouse
1
3
2
4
Case Solution
![Page 20: Solving the Unstructured Data Problem – Case Study the Unstructured Data Problem – Case Study Subodh Manicka, ... Federal Institute for Drugs and Medical ... Cognos BI Complaints](https://reader030.fdocuments.in/reader030/viewer/2022021505/5acc140f7f8b9a875a8c2a19/html5/thumbnails/20.jpg)
Solution Software Stack
• IBM Watson Explorer Advanced Edition (Content Analytics Components)
• IBM Datacap (Path #1)
• IBM Content Collector (Path #1)
• IBM Content Classification
• IBM PureData System (Path #2)
• IBM Case Manager (Path #3)
• Cognos BI (Path #4)
19
![Page 21: Solving the Unstructured Data Problem – Case Study the Unstructured Data Problem – Case Study Subodh Manicka, ... Federal Institute for Drugs and Medical ... Cognos BI Complaints](https://reader030.fdocuments.in/reader030/viewer/2022021505/5acc140f7f8b9a875a8c2a19/html5/thumbnails/21.jpg)
Logical Architecture of Solution
20
![Page 22: Solving the Unstructured Data Problem – Case Study the Unstructured Data Problem – Case Study Subodh Manicka, ... Federal Institute for Drugs and Medical ... Cognos BI Complaints](https://reader030.fdocuments.in/reader030/viewer/2022021505/5acc140f7f8b9a875a8c2a19/html5/thumbnails/22.jpg)
Enhancements with Watson Services
• Data Ingest Challenges
� Wide variety of formats – text, image, audio etc.
• Watson APIs
� Speech to Text
� Visual Recognition
� Language Translator
� Natural Language Classifier
� Natural Language Understanding
21
![Page 23: Solving the Unstructured Data Problem – Case Study the Unstructured Data Problem – Case Study Subodh Manicka, ... Federal Institute for Drugs and Medical ... Cognos BI Complaints](https://reader030.fdocuments.in/reader030/viewer/2022021505/5acc140f7f8b9a875a8c2a19/html5/thumbnails/23.jpg)
Potential Applications
• Potential areas of application
� Citizen Services
• Consumer product safety (Ex: CPSC)
• Automobile safety (Ex: NHTSA)
• Drug and Medical devices Safety (Ex: FDA)
• Health and Medical Research (Ex: NIH)
• Enterprise unstructured data processing
� Analyzing comments on regulations (Ex: FCC)
� Seized media analysis (Ex: Intel and Defense)
• Any place where volumes of unstructured or semi structured contents need to be analyzed for insight
22
![Page 24: Solving the Unstructured Data Problem – Case Study the Unstructured Data Problem – Case Study Subodh Manicka, ... Federal Institute for Drugs and Medical ... Cognos BI Complaints](https://reader030.fdocuments.in/reader030/viewer/2022021505/5acc140f7f8b9a875a8c2a19/html5/thumbnails/24.jpg)
23