thisistaimur/phenoembed-mitb2

PhenoEmbed MiT-B2

PhenoEmbed is a self-supervised temporal feature extractor for individual tree crowns observed in multispectral UAV image time series. It converts an aligned sequence of crown-centered multispectral crops into one L2-normalized, 256-dimensional vector summarizing seasonal crown appearance.

The model was developed to capture phenological changes such as leaf emergence, canopy closure, senescence, and leaf-off conditions. It is intended as a representation model for downstream tree-level Earth-observation tasks rather than as a species classifier or segmentation model.

Model Description

• Architecture: SegFormer MiT-B2 spatial encoder with temporal Transformer
• Base model: nvidia/mit-b2
• Input bands: red, green, red-edge, and near-infrared
• Temporal observations: 18 acquisition dates
• Crop extent: 16 m × 16 m per crown
• Stored crop resolution: 288 × 288 pixels
• Backbone input resolution: 224 × 224 pixels
• Output: one L2-normalized 256-dimensional embedding per crown
• Training method: self-supervised temporal contrastive learning and masked temporal reconstruction
• Spatial backbone: frozen during training
• Trainable components: multispectral adapter, temporal Transformer, projection head, and reconstruction head

A trainable 1 × 1 convolution maps the four multispectral bands to the three-channel input expected by the ImageNet-pretrained MiT-B2 backbone. It is initialized with red and green passed to their corresponding channels and the third channel initialized from the mean of red-edge and NIR. All adapter weights remain trainable.

Per-date spatial features are combined with normalized seasonal time features using a two-layer, four-head temporal Transformer.

Training Data

The model was trained on HeideBench, a multispectral UAV time-series dataset covering a forest patch in Dölauer Heide, Halle (Saale), Germany.

The training corpus contains:

• 18 UAV orthomosaics acquired between 6 March and 5 November 2025
• 5,885 crop-safe individual tree crowns
• 105,930 crown-date crop instances
• 5,297 training crowns and 588 validation crowns
• Canonical band order: R, G, RE, NIR
• Average source ground sampling distance: 5.53 cm per pixel

Crown polygons were used as fixed object anchors for extracting aligned crops of the same tree through time.

Input Preprocessing

Input reflectance values must be arranged in canonical order:

R, G, RE, NIR

The preprocessing used during training was:

1. Divide reflectance values by 10,000.
2. Clip values to the range [0, 4].
3. Normalize each band using the following statistics:

Band	Mean	Standard deviation
R	0.06105786	0.05220907
G	0.07939660	0.05130135
RE	0.31415916	0.20160384
NIR	0.67597259	0.40058157

Invalid pixels are excluded using the accompanying alpha mask.

Each acquisition date also receives a normalized seasonal coordinate:

(day - first_day) / (last_day - first_day)

Inputs from other sensors or sites should be calibrated to comparable reflectance units. Reusing the HeideBench normalization statistics outside the training domain may not be appropriate.

Training Configuration

• Batch size: 2
• Completed optimization steps: 50,000
• Optimizer: AdamW
• Learning rate: 5 × 10⁻⁵
• Weight decay: 10⁻⁴
• Temporal masking probability: 0.3
• Contrastive temperature: 0.2
• Contrastive-loss weight: 1.0
• Reconstruction-loss weight: 1.0
• Precision: mixed 16-bit
• Checkpoint selection: minimum validation objective

A separate batch-size-16 sensitivity run was also evaluated. It did not improve the intrinsic embedding diagnostics under its training schedule, but it used a different learning rate and stopping configuration and should not be interpreted as a controlled batch-size ablation.

Usage

PhenoEmbed uses a custom PyTorch Lightning architecture and cannot be loaded directly with transformers.AutoModel.

Clone and install the PhenoEmbed repository, prepare a compatible crop manifest, and run:

PYTHONPATH=src python -m phenoembed.inference.export_embeddings \
  --checkpoint-path phenoembed-mitb2-full.ckpt \
  --data-config configs/dataloader_full.toml \
  --output-path outputs/crown_embeddings.csv \
  --npz-path outputs/crown_embeddings.npz \
  --device cuda

The CSV contains the crown identifier, acquisition-date sequence, and 256 embedding dimensions. The optional NPZ output contains crown_id, date_sequence, and embedding arrays.

Evaluation

Intrinsic evaluation on 5,885 HeideBench crowns produced:

Diagnostic	Result
Variance explained by PC1 and PC2	25.1%
Variance explained by the first 8 PCs	71.8%
Median top-1 cosine similarity	0.946
Median cosine similarity among top-10 neighbors	0.902
NDVI-amplitude linear probe, five-fold CV R²	0.525
NDRE-amplitude linear probe, five-fold CV R²	0.414

The linear-probe results show that the embeddings retain measurable information about seasonal vegetation change. PCA and nearest-neighbor similarity are intrinsic representation diagnostics, not downstream accuracy measurements.

Intended Uses

PhenoEmbed is intended for:

• Crown-level temporal representation extraction
• Forest phenology analysis
• Similarity search and crown retrieval
• Phenology-aware feature generation
• Research on seasonally robust tree-level models
• Future integration with crown segmentation or classification systems

Limitations

• The model was trained on one forest site, one year, and one UAV sensor.
• Generalization across sites, years, sensors, and spatial resolutions has not yet been established.
• The model expects aligned observations of the same annotated crown through time.
• The selected model uses only a small number of in-batch contrastive negatives.
• The MiT-B2 backbone is frozen and receives four-band information through a learned four-to-three-channel adapter.
• The reconstruction objective predicts per-date band means rather than detailed spatial structure.
• Current evaluation is intrinsic. Improved downstream crown segmentation under seasonal shift has not yet been demonstrated.
• Embedding similarity must not be interpreted as species identity, ecological equivalence, health status, or segmentation accuracy without independent validation.

Citation

@inproceedings{khan2026phenoembed,
  title     = {PhenoEmbed: Self-Supervised Multispectral UAV Time-Series
               Embeddings for Individual Tree Crown Phenology},
  author    = {Khan, Taimur},
  year      = {2026},
  note      = {Resilience and AI Workshop at Informatik Festival 2026}
}

Please also cite the training dataset:

@dataset{khan2026heidebench,
  author    = {Khan, Taimur},
  title     = {HeideBench: A Multispectral UAV Time-Series Benchmark for
               Forest Crown Phenology in Dölauer Heide},
  publisher = {PANGAEA},
  year      = {2026},
  doi       = {10.1594/PANGAEA.993969}
}

License

The release license for the PhenoEmbed weights must be stated here before publication. Use of the model must also comply with the terms of the pretrained MiT-B2 model and the HeideBench dataset.