Selected Journal Papers

Abstract:

Machine learning has been leveraged for image analysis applications throughout a multitude of subspecialties. This position paper provides a perspective on the evolutionary trajectory of practical deep learning tools for genitourinary pathology through evaluating the most recent iterations of such algorithmic devices. Deep learning tools for genitourinary pathology demonstrate potential to enhance prognostic and predictive capacity for tumor assessment including grading, staging, and subtype identification, yet limitations in data availability, regulation, and standardization have stymied their implementation.

Journal of Pathology InformaticsVolume 14, 2023, 100177

Abstract:

Despite the recent progress in Deep Neural Networks (DNNs) to characterize histopathology images, compactly representing a gigapixel whole-slide image (WSI) via salient features to enable computational pathology is still an urgent need and a significant challenge. In this paper, we propose a novel WSI characterization approach to represent, search and classify biopsy specimens using a compact feature vector (CFV) extracted from a multitude of deep feature vectors. Since the non-optimal design and training of deep networks may result in many irrelevant and redundant features and also cause computational bottlenecks, we proposed a low-cost stochastic method to optimize the output of pre-trained deep networks using evolutionary algorithms to generate a very small set of features to accurately represent each tissue/biopsy. The performance of the proposed method has been assessed using WSIs from the publicly available TCGA image data. In addition to acquiring a very compact representation (i.e., 11,000 times smaller than the initial set of features), the optimized features achieved 93% classification accuracy resulting in 11% improvement compared to the published benchmarks. The experimental results reveal that the proposed method can reliably select salient features of the biopsy sample. Furthermore, the proposed approach holds the potential to immensely facilitate the adoption of digital pathology by enabling a new generation of WSI representation for efficient storage and more user-friendly visualization.

Abstract:

This paper investigates the effect of magnification on content-based image search in digital pathology
archives and proposes to use multi-magnification image representation. Image search in large archives of digital pathology slides provides researchers and medical professionals with an opportunity to match records of current and past patients and learn from evidently diagnosed and
treated cases. When working with microscopes, pathologists switch between different magnification levels while examining tissue specimens to find and evaluate various morphological features. Inspired by the conventional pathology workflow, we have investigated several magnification levels in digital pathology and their combinations to minimize the gap between AI-enabled image search methods and clinical settings. The proposed searching framework does not rely on any regional annotation and potentially applies to millions of unlabelled (raw) whole slide images. This paper suggests two approaches for combining magnification levels and compares their performance. The first approach obtains a single-vector deep feature representation for a digital slide, whereas the second approach
works with a multi-vector deep feature representation. We report the search results of 20×, 10×, and 5× magnifications and their combinations on a subset of The Cancer Genome Atlas (TCGA) repository. The experiments verify that cell-level information at the highest magnification is
essential for searching for diagnostic purposes. In contrast, low-magnification information may improve this assessment depending on the tumor type. Our multi-magnification approach achieved up to 11% F1-score improvement in searching among the urinary tract and brain tumor subtypes compared to the single-magnification image search.

Published in: IEEE Journal of Biomedical and Health Informatics ( Volume: 26, Issue: 9, September 2022)

Page(s): 4611 – 4622

Date of Publication: 10 June 2022 

ISSN Information:

PubMed ID: 35687644

INSPEC Accession Number: 22048046

DOI: 10.1109/JBHI.2022.3181531Publisher: IEEE

Abstract:

One of the main obstacles of adopting digital pathology is the challenge of efficient processing of hyperdimensional digitized biopsy samples, called whole slide images (WSIs). Exploiting deep learning and introducing compact WSI representations are urgently needed to accelerate image analysis and facilitate the visualization and interpretability of pathology results in a postpandemic world. In this article, we introduce a new evolutionary approach for WSI representation based on large-scale multiobjective optimization (LSMOP) of deep embeddings. We start with patch-based sampling to feed KimiaNet, a histopathology-specialized deep network, and to extract a multitude of feature vectors. Coarse multiobjective feature selection uses the reduced search space strategy guided by the classification accuracy and the number of features. In the second stage, the frequent features histogram (FFH), a novel WSI representation, is constructed by multiple runs of coarse LSMOP. Fine evolutionary feature selection is then applied to find a compact (short-length) feature vector based on the FFH and contributes to a more robust deep-learning approach to digital pathology supported by the stochastic power of evolutionary algorithms. We validate the proposed schemes using The Cancer Genome Atlas (TCGA) images in terms of WSI representation, classification accuracy, and feature quality. Furthermore, a novel decision space for multicriteria decision making in the LSMOP field is introduced. Finally, a patch-level visualization approach is proposed to increase the interpretability of deep features. The proposed evolutionary algorithm finds a very compact feature vector to represent a WSI (almost 14000 times smaller than the original feature vectors) with 8% higher accuracy compared to the codes provided by the state-of-the-art methods

Published in: IEEE Transactions on Evolutionary Computation ( Volume: 27, Issue: 1, February 2023)

Page(s): 52 – 66

Date of Publication: 26 May 2022 

ISSN Information:

INSPEC Accession Number: 22570371

DOI: 10.1109/TEVC.2022.3178299Publisher: IEEE

Abstract:

In the era of big data, methods for improving memory and computational efficiency have become crucial for the successful deployment of technology. Hashing is one of the most effective approaches to deal with the computational limitations associated with big data. One natural way to formulate this problem is spectral hashing, which directly incorporates an affinity to learning binary codes. However, owing to the binary constraints, the optimization becomes intractable. To mitigate this challenge, different relaxation approaches have been proposed to reduce the computational load required to obtain binary codes and still attain a good solution. The problem with all existing relaxation methods involves the use of one or more additional auxiliary variables to attain high-quality binary codes while relaxing the problem. The existence of auxiliary variables leads to the coordinate descent approach, which increases the computational complexity. We argue that the introduction of these variables is unnecessary. To this end, we propose a novel relaxed formulation for spectral hashing that adds no additional variables to the problem. Furthermore, instead of solving the problem in the original space where the number of variables is equal to the data points, we solve the problem in a much smaller space and retrieve the binary codes from this solution. This technique reduces both the memory and computational complexity simultaneously. We apply two optimization techniques, namely, the projected gradient and optimization on the manifold, to obtain the solution. Using comprehensive experiments on four public datasets, we show that the proposed efficient spectral hashing (ESH) algorithm achieves a highly competitive retrieval performance compared with the state-of-the-art algorithms at low complexity.

MSC 41A05 41A10 65D05 65D17

https://doi.org/10.1016/j.cviu.2022.103415 Get rights and content

Abstract:

The artificial intelligence revolution has been spurred forward by the availability of large-scale datasets. In contrast, the paucity of large-scale medical datasets hinders the application of machine learning in healthcare. The lack of publicly available multi-centric and diverse datasets mainly stems from confidentiality and privacy concerns around sharing medical data. To demonstrate a feasible path forward in medical image imaging, we conduct a case study of applying a differentially private federated learning framework for analysis of histopathology images, the largest and perhaps most complex medical images. We study the effects of IID and non-IID distributions along with the number of healthcare providers, i.e., hospitals and clinics, and the individual dataset sizes, using The Cancer Genome Atlas (TCGA) dataset, a public repository, to simulate a distributed environment. We empirically compare the performance of private, distributed training to conventional training and demonstrate that distributed training can achieve similar performance with strong privacy guarantees. We also study the effect of different source domains for histopathology images by evaluating the performance using external validation. Our work indicates that differentially private federated learning is a viable and reliable framework for the collaborative development of machine learning models in medical image analysis.

Abstract:

Image analysis in digital pathology has proven to be one of the most challenging fields in medical imaging for AI-driven classification and search tasks. Due to their gigapixel dimensions, whole slide images (WSIs) are difficult to represent for computational pathology. Self-supervised learning (SSL) has recently demonstrated excellent performance in learning effective representations on pretext objectives, which may improve the generalizations of downstream tasks. Previous self-supervised representation methods rely on patch selection and classification such that the effect of SSL on end-to-end WSI representation is not investigated. In contrast to existing augmentation-based SSL methods, this paper proposes a novel self-supervised learning scheme based on the available primary site information. We also design a fully supervised contrastive learning setup to increase the robustness of the representations for WSI classification and search for both pretext and downstream tasks. We trained and evaluated the model on more than 6000 WSIs from The Cancer Genome Atlas (TCGA) repository provided by the National Cancer Institute. The proposed architecture achieved excellent results on most primary sites and cancer subtypes. We also achieved the best result on validation on a lung cancer classification task.

Abstract:

An effective unsupervised hashing algorithm leads to compact binary codes preserving the neighborhood structure of data as much as possible. One of the most established schemes for unsupervised hashing is to reduce the dimensionality of data and then find a rigid (neighborhood-preserving) transformation that reduces the quantization error. Although employing rigid transformations is effective, we may not reduce quantization loss to the ultimate limits. As well, reducing dimensionality and quantization loss in two separate steps seems to be sub-optimal. Motivated by these shortcomings, we propose to employ both rigid and non-rigid transformations to reduce quantization error and dimensionality simultaneously. We relax the orthogonality constraint on the projection in a PCA-formulation and regularize this by a quantization term. We show that both the non-rigid projection matrix and rotation matrix contribute towards minimizing quantization loss but in different ways. A scalable nested coordinate descent approach is proposed to optimize this mixed-integer optimization problem. We evaluate the proposed method on five public benchmark datasets providing almost half a million images. Comparative results indicate that the proposed method mostly outperforms state-of-art linear methods and competes with end-to-end deep solutions.

MSC 41A05 41A10 65D05 65D17

https://doi.org/10.1016/j.patrec.2021.11.007

Abstract:

Although deep learning networks applied to digital images have shown impressive results for many
pathology-related tasks, their black-box approach and limitation in terms of interpretability are sig-
nificant obstacles for their widespread clinical utility. This study investigates the visualization of deep
features (DFs) to characterize two lung cancer subtypes, adenocarcinoma and squamous cell carcinoma.
It demonstrates that a subset of DFs, called prominent DFs, can accurately distinguish these two cancer
subtypes. Visualization of such individual DFs allows for a better understanding of histopathologic
patterns at both the whole-slide and patch levels, and discrimination of these cancer types. These DFs
were visualized at the whole slide image level through DF-specific heatmaps and at tissue patch level
through the generation of activation maps. In addition, these prominent DFs can distinguish carcinomas
of organs other than the lung. This framework may serve as a platform for evaluating the interpretability
of any deep network for diagnostic decision making.

Abstract:

One of the major obstacles in reaching diagnostic consensus is observer variability. With the recent
success of artificial intelligence, particularly the deep networks, the question emerges as to whether
the fundamental challenge of diagnostic imaging can now be resolved. This article briefly reviews
the problem and how eventually both supervised and unsupervised AI technologies could help to
overcome it.

Am J Pathol 2021, 191: 1702e1708; https://doi.org/10.1016/j.ajpath.2021.01.015

The American Journal of Pathology Volume 191, Issue 10, October 2021, Pages 1702-1708

Abstract:

Background

Pathology synopses consist of semi-structured or unstructured text summarizing visual information by observing human tissue. Experts write and interpret these synopses with high domain-specific knowledge to extract tissue semantics and formulate a diagnosis in the context of ancillary testing and clinical information. The limited number of specialists available to interpret pathology synopses restricts the utility of the inherent information. Deep learning offers a tool for information extraction and automatic feature generation from complex datasets.

Methods

Using an active learning approach, we developed a set of semantic labels for bone marrow aspirate pathology synopses. We then trained a transformer-based deep-learning model to map these synopses to one or more semantic labels, and extracted learned embeddings (i.e., meaningful attributes) from the model’s hidden layer.

Results

Here we demonstrate that with a small amount of training data, a transformer-based natural language model can extract embeddings from pathology synopses that capture diagnostically relevant information. On average, these embeddings can be used to generate semantic labels mapping patients to probable diagnostic groups with a micro-average F1 score of 0.779 Â ± 0.025.

Conclusions

We provide a generalizable deep learning model and approach to unlock the semantic information inherent in pathology synopses toward improved diagnostics, biodiscovery and AI-assisted computational pathology.

Abstract:

Fast diagnosis and treatment of pneumothorax, a collapsed or dropped lung, is crucial to avoid fatalities. Pneumothorax is typically detected on a chest X-ray image through visual inspection by experienced radiologists. However, the detection rate is quite low due to the complexity of visual inspection for small lung collapses. Therefore, there is an urgent need for automated detection systems to assist radiologists. Although deep learning classifers generally deliver high accuracy levels in many applications, they may not be useful in clinical practice due to the lack of high-quality and representative labeled image sets. Alternatively, searching in the archive of past cases to fnd
matching images may serve as a “virtual second opinion” through accessing the metadata of matched evidently diagnosed cases. To use image search as a triaging or diagnosis assistant, we must frst tag
all chest X-ray images with expressive identifers, i.e., deep features. Then, given a query chest X-ray image, the majority vote among the top k retrieved images can provide a more explainable output. In this study, we searched in a repository with more than 550,000 chest X-ray images. We developed the Autoencoding Thorax Net (short AutoThorax -Net) for image search in chest radiographs.
Experimental results show that image search based on AutoThorax -Net features can achieve high identifcation performance providing a path towards real-world deployment. We achieved 92% AUC
accuracy for a semi-automated search in 194,608 images (pneumothorax and normal) and 82% AUC accuracy for fully automated search in 551,383 images (normal, pneumothorax and many other chest diseases).

Abstract:

Feature vectors provided by pre-trained deep artificial neural networks have become a dominant source for image representation in recent literature. Their contribution to the performance of image analysis can be improved through fine-tuning. As an ultimate solution, one might even train a deep network from scratch with the domain-relevant images, a highly desirable option which is generally impeded in pathology by lack of labeled images and the computational expense. In this study, we propose a new network, namely KimiaNet, that employs the topology of the DenseNet with four dense blocks, fine-tuned and trained with histopathology images in different configurations. We used more than 240,000 image patches with 1000×1000 pixels acquired at 20× magnification through our proposed “high-cellularity mosaic” approach to enable the usage of weak labels of 7126 whole slide images of formalin-fixed paraffin-embedded human pathology samples publicly available through The Cancer Genome Atlas (TCGA) repository. We tested KimiaNet using three public datasets, namely TCGA, endometrial cancer images, and colorectal cancer images by evaluating the performance of search and classification when corresponding features of different networks are used for image representation. As well, we designed and trained multiple convolutional batch-normalized ReLU (CBR) networks. The results show that KimiaNet provides superior results compared to the original DenseNet and smaller CBR networks when used as feature extractor to represent histopathology images.

Medical Image Analysis Volume 70, May 2021, 102032

Abstract:

In computer science, textbooks talk about the “garbage in,
garbage out” concept (GIGO); i.e., low-quality input data generates
unreliable output or “garbage.” GIGO becomes, even
more, a pressing issue when we are dealing with highly complex
data modalities, such as radiographs and computed tomography
scans.

European Radiology volume 31, pages3553–3554 (2021)

Abstract:

Context.—

Pathologists may encounter extraneous pieces of tissue (tissue floaters) on glass slides because of specimen cross-contamination. Troubleshooting this problem, including performing molecular tests for tissue identification if available, is time consuming and often does not satisfactorily resolve the problem.

Objective.—

To demonstrate the feasibility of using an image search tool to resolve the tissue floater conundrum.

Design.—

A glass slide was produced containing 2 separate hematoxylin and eosin (H&E)-stained tissue floaters. This fabricated slide was digitized along with the 2 slides containing the original tumors used to create these floaters. These slides were then embedded into a dataset of 2325 whole slide images comprising a wide variety of H&E stained diagnostic entities. Digital slides were broken up into patches and the patch features converted into barcodes for indexing and easy retrieval. A deep learning-based image search tool was employed to extract features from patches via barcodes, hence enabling image matching to each tissue floater.

Results.—

There was a very high likelihood of finding a correct tumor match for the queried tissue floater when searching the digital database. Search results repeatedly yielded a correct match within the top 3 retrieved images. The retrieval accuracy improved when greater proportions of the floater were selected. The time to run a search was completed within several milliseconds.

Conclusions.—

Using an image search tool offers pathologists an additional method to rapidly resolve the tissue floater conundrum, especially for those laboratories that have transitioned to going fully digital for primary diagnosis.

Arch Pathol Lab Med (2021) 145 (3): 359–364.

https://doi.org/10.5858/arpa.2020-0034-OA

Abstract:

Within artificial intelligence and machine learning, a generative model is a powerful tool for learning any kind of datadistribution. With the advent of deep learning and its success in image recognition, thefield of deep generativemodels has clearly emerged as one of the promisingfields for medical imaging. In a recent issue ofTheJournalofPathology, Levine, Pengetaldemonstrate the ability of generative models to synthesize high-quality pathologyimages. They suggested that generative models can serve as an unlimited source of images either for educating fresh-man pathologists or training machine learning models for diverse image analysis tasks, especially in scarce cases,while resolving patients’privacy and confidentiality concerns.© 2020 The Pathological Society of Great Britain and Ireland. Published by John Wiley & Sons, Ltd.

Abstract:
With the emergence of digital pathology, searching for similar images in large archives has gained considerable attention. Image retrieval can provide pathologists with unprecedented access to the evidence embodied in already diagnosed and treated cases from the past. This paper proposes a search engine specialized for digital pathology, called Yottixel, a portmanteau for “one yotta pixel,” alluding to the big-data
nature of histopathology images. The most impressive characteristic of Yottixel is its ability to represent whole slide images (WSIs) in a compact manner. Yottixel can perform millions of searches in real-time with a high search accuracy and low storage profile. Yottixel uses an intelligent indexing algorithm capable of representing WSIs with a mosaic of patches which are then converted into barcodes, called “Bunch
of Barcodes” (BoB), the most prominent performance enabler of Yottixel. The performance of the prototype platform is qualitatively tested using 300 WSIs from the University of Pittsburgh Medical Center (UPMC) and 2,020 WSIs from The Cancer Genome Atlas Program (TCGA) provided by the National Cancer Institute. Both datasets amount to more than 4,000,000 patches of 1000 × 1000 pixels. We report three sets of experiments that show that Yottixel can accurately retrieve organs and malignancies, and its semantic ordering shows good agreement with the subjective evaluation of human observers.

https://doi.org/10.1016/j.media.2020.101757

Abstract:

The colorless biopsied tissue samples are usually stained in order to visualize different microscopic structures for diagnostic purposes. But color variations associated with the process of sample preparation, usage of raw materials, diverse staining protocols, and using different slide scanners may adversely influence both visual inspection and computer-aided image analysis. As a result, many methods are proposed for histopathology image stain normalization in recent years. In this study, we introduce a novel approach for stain normalization based on learning a mixture of multivariate skew-normal distributions for stain clustering and parameter estimation alongside a stain transformation technique. The proposed method, labeled “Class-Agnostic Weighted Normalization” (short CLAW normalization), has the ability to normalize a source image by learning the color distribution of both source and target images within an expectation-maximization framework. The novelty of this approach is its flexibility to quantify the underlying both symmetric and nonsymmetric distributions of the different stain components while it is considering the spatial information. The performance of this new stain normalization scheme is tested on several publicly available digital pathology datasets to compare it against state-ofthe-art normalization algorithms in terms of ability to preserve the image structure and information. All in all, our proposed method performed superior more consistently in comparison with existing methods in terms of information preservation, visual quality enhancement, and boosting computer-aided diagnosis algorithm performance.

IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 39, NO. 11, NOVEMBER 2020

Abstract:

The emergence of digital pathology has opened new horizons for histopathology. Artificial intelligence (AI) algorithms are able to operate on digitized slides to assist pathologists with different tasks. Whereas AI-involving classification and segmentation methods have obvious benefits for image analysis, image search represents a fundamental shift in computational pathology. Matching the pathology of new patients with already diagnosed and curated cases offers pathologists a new approach to improve diagnostic accuracy through visual inspection of similar cases and computational majority vote for consensus building. In this study, we report the results from searching the largest public repository (The Cancer Genome Atlas, TCGA) of whole-slide images from almost 11,000 patients. We successfully indexed and searched almost 30,000 high-resolution digitized slides constituting 16 terabytes of data comprised of 20 million 1000 × 1000 pixels image patches. The TCGA image database covers 25 anatomic sites and contains 32 cancer subtypes. High-performance storage and GPU power were employed for experimentation. The results were assessed with conservative “majority voting” to build consensus for subtype diagnosis through vertical search and demonstrated high accuracy values for both frozen section slides (e.g., bladder urothelial carcinoma 93%, kidney renal clear cell carcinoma 97%, and ovarian serous cystadenocarcinoma 99%) and permanent histopathology slides (e.g., prostate adenocarcinoma 98%, skin cutaneous melanoma 99%, and thymoma 100%). The key finding of this validation study was that computational consensus appears to be possible for rendering diagnoses if a sufficiently large number of searchable cases are available for each cancer subtype.

npj Digital Medicine volume 3, Article number: 31 (2020) 

Abstract:

In light of the recent success of artificial intelligence (AI) in computer vision applications, many researchers and physicians expect that AI would be able to assist in many tasks in digital pathology. Although opportunities are both manifest and tangible, there are clearly many challenges that need to be overcome in order to exploit the AI potentials in computational pathology. In this paper, we strive to provide a realistic account of all challenges and opportunities of adopting AI algorithms in digital pathology from both engineering and pathology perspectives.

Abstract:
With the availability of whole-slide imaging in pathology, high-resolution images offer a more convenient disease observation but also require content-based retrieval of large scans. The bag-of-visual-words methodology has shown a high ability to describe the image content for recognition and retrieval purposes. In this work, a variant of the bag-of-visual-words with multiple dictionaries for histopathology image classification is proposed and tested on the image dataset Kimia Path24 with more than 27,000 patches of size 1000 × 1000 belonging to
24 different classes. Features are extracted from patches and clustered to form multiple codebooks. The histogram intersection approach and support vector machines are exploited to build multiple classifiers. At last, the majority voting determines the final classification for each patch. The experiments demonstrate the superiority of the proposed method for histopathology images that surpasses deep networks, LBP and other BoW results.