Skip to the content.

Welcome to the SEE-Insight Research Team

SEE-Insight logo

The SEE-Insight Research Team is lead by Dr. Dirk Colbry in the Department of Computational Mathematics, Science and Engineering at Michigan State University.

We hold a weekly group meeting on Tuesdays 3:00-4:00pm ET. Please email Dr. Colbry if you would like to join us and he can send you the zoom coordinates and/or add you to our group email list.

Problem Description

Low-cost imaging allow researchers who rely on visual observations to digitally record experiments, resulting in huge databases of images that can be reviewed and re-reviewed over time. “Scientific image understanding” is the process of extracting scientific measurements out of images. Since the information of interest within an image changes with each new research question there is no single “universal measurement” and no single software program that can analyze every image or solve every problem. Instead, researchers must manually re-annotate images or write new analysis software every time they want to ask new questions.

Machine learning (ML) offers a way to “automatically” find a customized image analysis algorithm for new research workflows. However, traditional supervised ML approaches (e.g., Artificial Neural Networks) require large datasets of pre-annotated images for training, which creates a circular problem: researchers must manually annotate their images in order to create a training dataset so that traditional ML approaches can find an algorithm to automatically annotate their images. This can be feasible for large, well-funded projects and domains (e.g., Medical Imaging, Self-Driving Cars) but not for smaller, exploratory projects where researchers want to use their data to test simple hypotheses or ask questions that have never been studied before. During this early stage of the scientific process, which we are calling “Exploratory Image Understanding,” it is common to manually annotate image and video frame-by-frame, which is an extremely slow process subject to variations in quality and detail.

SEE-Insight Approach

The SEE-Insight Team is developing image understanding tools which focus on common workflows used in scientific image understanding for applications in engineering, medical imaging, biology, etc. The goal is to develop tools that are more than just a manual annotation system. As the scientist annotates their images, the tools will take their annotations (starting from the very first image) and use machine learning to search the “algorithm space” to try and identify candidate algorithms based on their specified workflow. If a good candidate algorithm is found, then the tools will start making suggestions to the researcher. In the worst case, using just the tools will take no more time than scientists would need to annotate the images manually; the result will be an annotated dataset which they can use to conduct their science or to feed into a more traditional ML system. However, in the best case, a good candidate algorithm can be automatically identified, that can help speed up some, or all, of the annotation process, saving researcher time and speeding up their overall research workflow.

The SEE-Insight Project is researching and developing tools using the following design guidelines:

Summary video about SEE-Insight Research

The SEE-Tools