Visual Recognition of Tissue Patterns on Pathology Slides

web-pathology.net

FOR HEALTHCARE AND DRUG DISCOVERY

SMART IMAGING TECHNOLOGIES

Personalized Cancer Therapy: Knowledge Path

Personalized Therapy

Medical History and

Personal Information

Genetic Information

Pathology Information

(tumor biomarkers)

Personalized cancer therapy is a treatment strategy centered on the ability to predict which patients are more likely to respond to specific cancer therapies.

This approach is founded upon the idea that tumor biomarkers are associated with patient prognosis and tumor response to therapy.

In addition, patient genetic factors can be associated with drug metabolism, drug response and drug toxicity.

Personalized tumor molecular profiles, tumor disease site and other patient characteristics are then potentially used for determining optimum individualized therapy options.

Source: MD Anderson Cancer Center

Pathology, the “study of disease”, is an essential component for analysis of personalized cancer therapy options

Needs for Healthcare

FOR HEALTHCARE

Diagnostic Pattern Library: Applications and Benefits

Classified cancer pattern library is valuable digital asset that can be licensed to other parties to train visual recognition and image analysis algorithms.

Visual recognition application can be used to automatically annotate digital pathology slides and link them with the rest of institutional cancer knowledge base. This application can be licensed to third parties to use for the same purposes.

Research and Clinical Applications:• Computer-assisted cancer diagnosis with pre-screening,

suggestive diagnosis options and contextual links to cancer knowledge libraries (similar cases, experts, research, additional tests etc.)

• Data mining and of advanced analytics of historic tissue samples for cancer patients with known outcomes with purpose of building predictive knowledge bases for cancer care and drug discovery

FOR HEALTHCARE

Researchers need to measure drug effects on tissue.

They also need to analyze relationships between effect on tissue and all other relevant data in the drug research project.

Drug effects on tissue are expressed via biomarker patterns on tissue slides. Researchers need to turn visual pattern expression into usable data.

They need technology to:• Objectify pattern definitions to deal with subjective biases of human

observers• Search for similar biomarker expression patterns across archives• Aggregate, analyze and organize pattern expression information• Link pattern information to other “omnics” and project research data

Needs for Drug Discovery

FOR DRUG DISCOVERY

Utilizing Pathology Knowledge: The Challenge

Traditionally, pathology diagnosis is presented in descriptive natural language statements. Often It is verbose professional opinion of human expert with little quantitative information Statistical agreement between human experts is 75%-85%.

Pathology diagnosis is rendered by pathologist observing patterns of cells on tissue slide under the microscope. In order to be useful for comparison and analysis these observations must be:

• Quantified• Objectified

This can be achieved (in theory) by digitizing pathology slide and applying image analysis algorithms to quantify cell pattern expressions.

Analyzing Pathology Slides: The Real Life Issues

In practice traditional image analysis approach often fall short of expectations for number of reasons:

• Target image features are complex and difficult for engineers to formalize and code.

• Algorithm development process is inherently complicated: only pathologists know that patterns they need but only image analysis engineers may know how to extract them

• Tissues have variability and algorithms developed on one sample may not work on others. Algorithms have to be redesigned by engineers every time new “out of range” samples are encountered

• Digital tissue slides are very large and processing every pixel is extremely computationally expensive, making “online” analysis impossible or impractical

Analyzing Pathology Slides: Machine Learning

Machine Learning Neural Networks learn to recognize images in the same way humans do – by example, rather than by formalized “handcrafted features”.

Since 2012 major improvement in visual recognition was achieved with so called deep learning neural networks. Latest generation of Visual Recognition Neural Networks achieve accuracy of recognition of natural objects similar to human observers. This area of technology is experiencing explosive growth.

Using Machine Learning brings number of advantages to visual recognition applications:• No need to formalize complex “handcrafted features”, pathologist

can just point to patterns they need to recognize• No dependency on image analysis engineers (almost)• System can be trained on very large number of samples to achieve

robust recognition• New data samples can be added to model easily to increase

accuracy

Machine Learning: RequirementsMachine Learning approach to pattern recognition creates new functional requirements for digital pathology software

• Robust visual recognition models need large number of training images which requires more time for annotating than single pathologist can provide. This problem can be solved by utilizing number of pathologists creating annotations for training visual recognition models

Collaborative Model Training (Crowdsourcing)

• Digital Pathology system should have capability for extracting specially formatted image data sets on demand for training neural networks

Training Data Extraction

• Training of Neural Network requires massive parallel GPU computing power for a short time. This scalable computing power can be economically delivered by scalable cloud infrastructures such as IBM, Amazon or Microsoft.

Cloud Deployment

• Digital pathology software should be able to send imaging data to neural network application for recognition and visualize responses for user.

API Integration and Visualization Interface

The Solution: Pattern Recognition with Machine Learning

• Last generation deep learning convolution networks can identify target tissue patters with 95% accuracy

Deep Learning

• Pathologists can train recognition solution by simply annotating target tissue patters on slides in their workspace

• They can easily set up classes of patterns for identification

Easy Training

• Robust solutions can be trained from multiple slides to identify target tissue patterns reliably across large variety of samples

• Recognition models can be retrained easily if new patterns or different samples should be added

Robust Recognition Models

• Slides in digital archives can be processed automatically for pattern detection and labeled based on findings

• New slides can be analyzed and classified on upload with suggestive classification available when human expert opens the slide

Automatic Processing

• Visualization overlays help quickly locate and review target patterns• Visualization layer provides quantitative information about patterns

Advanced Visualization

• All data is stored in the database and available for search, data mining and analytics Powerful Analytics

Our software can train neural networks and utilize latest deep learning visual recognition solutions from best in class solution providers

The Solution: How it works

• Human experts mark areas with target patterns on digital slides in the web interface• Multiple people across geographic locations can work on the same slide library at the

same time• Software automatically prepares and extracts imaging data for training visual recognition

model in appropriate format• Massive amount of training data can be produced quickly and efficiently with

crowdsourcing approach

1. Preparation of Training Data

• We train visual recognition models using proven templates and best-in-class machine learning service providers

2. Training Visual Recognition Model

• Tissues on new slides coming to server are automatically recognized against known taxonomy and labeled according to application logic.

3. Automatic Recognition Processing

• Users can mark the area and search for similar patterns on the same slide or across entire slide archive

4. Search by Example

• Software shows identified patterns on slides using interactive “smart” overlays and computes statistics

5. Viewing Results

• Users can correct visual recognition results by marking area with appropriate label. New data will be added to the training set to improve recognition

6. Continuous Learning

Integration: Data Mining and Discovery

• Non SQL flexible indexed database architecture allows integrated storage of different data items across multiple locations

Distributed Database

• Flexible structure allows storing and integrating various data in the single information store

• New data can be added to database structure at any time

Comprehensive Data

• Selection and navigation is possible for any data item in the database• Global search on any data is instant even for millions of items

Instant Search and Navigation

• Data items can be linked with external data sources and knowledge bases such as diagnostic codes, SNOMED classifications or proprietary knowledge bases

Data Linking

We provide instant search, navigation and data mining ability across millions of slides