Wheat image-text biocuration

Curator-supervised multimodal image-text learning for wheat disease phenotype evidence from published leaf assay images.

Project summary

This repository supports a small proof-of-concept study based on literature-derived wheat leaf assay images and associated figure text. The aim is to explore whether a multimodal image-text model can help classify wheat disease severity and support phenotype evidence extraction for biocuration.

The workflow combines:

• an image branch using a pre-trained GoogLeNet model;
• a text branch using BERT-base-uncased;
• feature-level fusion of image and text embeddings;
• four disease severity classes: low, high, medium, and none;
• stratified five-fold cross-validation;
• image and text augmentation;
• weighted sampling and regularisation.

Authorship and code credit

Original code generated by:

Saman Maghsoudi, Liverpool Hope University

This repository has been annotated and refactored to support publication, reproducibility and public sharing. AI coding tools were used to assist with repository setup and upload; all content remains under author review and responsibility.

Important limitation

This repository supports a small proof-of-concept study based on 40 literature-derived image-caption samples from six publications. The model is not a deployable diagnostic system and should not be used for field, glasshouse, or clinical diagnosis.

The results should be interpreted as evidence that multimodal image-text learning may be useful for curator-supervised phenotype evidence extraction, not as proof of robust diagnostic performance.

Suggested repository structure

wheat-leaf-image-text-biocuration/
├── README.md
├── requirements.txt
├── .gitignore
├── src/
│   └── train_multimodal_model.py
├── data/
│   └── README.md
└── outputs/
    └── README.md

Expected input data

The script expects:

1. An Excel metadata file with two sheets:

   • Figures
   • Captions

2. A folder containing the extracted assay images.

The Figures sheet should include at least:

• pmid
• filename
• assay_label
• label

The Captions sheet should include at least:

• pmid
• caption

The script merges the two sheets using pmid. It then combines assay_label and caption to create the text input.

Installation

Create a fresh Python environment, then install dependencies.

pip install -r requirements.txt
python -m nltk.downloader wordnet omw-1.4

A GPU is recommended but not required. On a CPU the run may be slow.

Example run

python src/train_multimodal_model.py \
  --excel-file data/zymo_labelled_data.xlsx \
  --image-base-dir data/phibase_assay_images \
  --output-dir outputs

On Windows PowerShell, use backticks instead of backslashes for line continuation:

python src/train_multimodal_model.py `
  --excel-file data/zymo_labelled_data.xlsx `
  --image-base-dir data/phibase_assay_images `
  --output-dir outputs

Main outputs

The script writes outputs to the selected output folder:

• cross_validation_results.json
• combined_confusion_matrix_bert_entity.png
• best_overall_model_bert_entity.pth

Notes for manuscript interpretation

This code uses full BERT fine-tuning by default. The small dataset size means that overfitting remains possible despite augmentation, weighted sampling, dropout, batch normalisation, label smoothing and cross-validation.

The cross-validation results should be presented as proof of concept. Independent testing on unseen publications would be required before making claims about generalisability.

Citation

Citation details can be added after the manuscript or preprint is available.