BioScan-CLIP: Contrastive Learning forAligning Biological Images and DNA Sequences
152
by ZeMing Gong and Austin T. Wang
Published onMay 27, 2024
BioScan-CLIP: Contrastive Learning forAligning Biological Images and DNA Sequences
·
Abstract
Measuring biodiversity is crucial for understanding ecosystem health. Whilst prior works have developed machine learning models for taxonomic classification of photographic images and DNA separately, in this work, we introduce a multimodal approach combining both, employing CLIP's methodology to align images, DNA barcodes, and textual data in a unified embedding space. This allows for accurate classification of both known and unknown insect species without task-specific fine-tuning, leveraging contrastive learning for the first time to fuse DNA and image data. Our method surpasses previous single-modality approaches in accuracy by over 11\% on zero-shot learning tasks, showcasing its effectiveness in biodiversity studies.
Creative Commons Attribution 4.0 International License (CC-BY 4.0)