✨🌍 GeoChrono✨: Benchmarking and Rethinking Long-Term Temporal Understanding in Remote Sensing

🔥 Accepted at ACM MM 2026 🔥

Earth's surface undergoes continuous long-term transformation, yet existing interpretation methods largely confine themselves to static snapshots or bi-temporal comparisons, leaving the rich temporal trajectory of landscape evolution untapped. This repository provides the full stack for studying long-term temporal understanding in remote sensing:

• 📊 ChronoBench — a four-level cognitive benchmark (Land Cover Perception → Temporal Recognition → Long-Term Memory → Spatio-Temporal Reasoning) with 12 sub-tasks and 17,689 rigorously validated QA pairs over 3,469 high-resolution images spanning 500 regions across 39 major U.S. cities.
• 📚 ChronoInstruct — a large-scale instruction-tuning dataset with 104,949 QA pairs in three answer formats (multiple-choice, short-text, and free-form natural language).
• 🤖 GeoChrono — an MLLM tailored for long-term remote sensing interpretation, built on Qwen3-VL-4B-Instruct with a Temporal Trajectory Encoder (TempEnc) and a Coarse-to-Fine Token Compressor (C2FComp), achieving 78.34% overall accuracy on ChronoBench — surpassing the leading commercial MLLMs by over 20%.

📢 News

• [2026/08] 🧩 ChronoBench can now be evaluated with lmms-eval — --tasks chronobench runs all 19 sub-tasks on any backend it supports. The task pack ships in lmms_eval_tasks/ and is loaded with --include_path (contributing it upstream is planned); its answer parsing and prompt content are verbatim ports of the reference harness (verified on 484,736 recorded predictions and all 17,689 prompts), and end-to-end scores agree to within measurement noise. The existing test/ workflow is unchanged — this is a second, optional entry point. See 🧩 Evaluating with lmms-eval.
• [2026/08] 🗂️ Added a task reference mapping the benchmark files to the sub-task names of the paper. The task files carry the working names they were built under (e.g. Change_Speed_Substantial.json is CCME), which are not self-explanatory, so the mapping is now documented here and written into every evaluation result file. This is additive only — no file, name, or QA pair changed, and existing downloads, scripts, and results stay valid. See 🗂️ Task Reference.
• [2026/07] 🔥🔥🔥 GeoChrono is accepted at ACM MM 2026!
• [2026/07] 🎉 ChronoBench & ChronoInstruct released on Hugging Face and ModelScope!
• [2026/07] 🎉 GeoChrono model weights (LoRA + TempEnc) released on Hugging Face and ModelScope!
• [2026/07] 🎉 Code for benchmark construction, evaluation, and training is open-sourced in this repository.

🌟 Overview

🔹 ChronoBench: a four-level cognitive hierarchy

ChronoBench decomposes long-term remote sensing understanding into four progressively demanding competency levels, each framed by a core question:

Level	Competency	Core Question
L0	Land Cover Perception	What is here at a given time?
L1	Temporal Recognition	What changed between specified time points?
L2	Long-Term Memory	When did changes occur, and what is the complete evolution history?
L3	Spatio-Temporal Reasoning	What can be inferred by comparing across locations and time?

All QA pairs are produced by a fully rule-based pipeline grounded in human-annotated semantic change masks, followed by a two-stage human quality-control process. See data_construction/ for the complete construction code and its README for the per-task generation logic.

🔹 GeoChrono: temporal trajectory modeling

GeoChrono leverages the physical prior that each geographic parcel remains spatially fixed while its semantics evolve. TempEnc decouples the spatio-temporal feature volume into per-location temporal trajectories through three stages — Spatial Context Aggregation, Hybrid Temporal Attention (bidirectional + causal), and text-guided Semantic Focusing. C2FComp exploits the strong foreground-background imbalance of remote sensing imagery: prompt-guided saliency selectively preserves full-resolution fine tokens for task-relevant regions while condensing the static background into compact coarse tokens, reducing visual tokens by over 56% while retaining 94.6% of the full model's performance.

🗂️ Task Reference

The benchmark files kept the working names they were built under, so a few of them do not reveal which sub-task of the paper they implement — Change_Speed_Substantial.json is CCME, and Bi_Temporal_Class_Level_Change_Perception.json / ..._Single.json are two different sub-tasks (BCC, multiple choice, and BCAC, single choice). The table below is the authoritative mapping; the same information is emitted into every evaluation result file, so metrics.json can be read straight against the results table of the paper.

Nothing was renamed. File names, data, and metrics are exactly as originally released. If you downloaded ChronoBench or ChronoInstruct before, there is nothing to re-download and nothing to change in your scripts.

19 benchmark files → 12 sub-tasks → 4 competency levels

Level	Abbr.	Sub-task (paper)	Localization	Format	# QA	Benchmark file
L0	OP	Object Perception	Geo Coordinate	Single Choice	665	Object_Classification_Coordinate.json
L0	OP	Object Perception	HBB Box	Single Choice	665	Object_Classification_HBB_Box.json
L1	BCAC	Bi-Temporal Class-Level Area Change Recognition	Image	Single Choice	3,914	Bi_Temporal_Class_Level_Change_Perception_Single.json
L1	BCC	Bi-Temporal Class-Level Change Recognition	Image	Multiple Choice	1,314	Bi_Temporal_Class_Level_Change_Perception.json
L1	BOC	Bi-Temporal Object-Level Change Recognition	Geo Coordinate	Multiple Choice	1,795	Temporal_Object_Level_Change_Perception_Geo_Coordinate.json
L1	BOC	Bi-Temporal Object-Level Change Recognition	HBB Box	Multiple Choice	1,835	Temporal_Object_Level_Change_Perception_HBB_Box.json
L1	LCC	Long-Temporal Class-Level Change Recognition	Image	Multiple Choice	737	Long_Temporal_Class_Level_Change_Perception.json
L2	OAM	Object Appear Memory	Geo Coordinate	Single Choice	1,103	Object_Appear_Memory_Coordinate.json
L2	OAM	Object Appear Memory	HBB Box	Single Choice	1,103	Object_Appear_Memory_HBB_Box.json
L2	OCM	Object Change Memory	Geo Coordinate	Single Choice	755	Object_Change_Memory_Coordinate.json
L2	OCM	Object Change Memory	HBB Box	Single Choice	755	Object_Change_Memory_HBB_Box.json
L2	OHM	Object History Memory	Geo Coordinate	Ordered Sequence	440	Object_History_Memory_Coordinate.json
L2	OHM	Object History Memory	HBB Box	Ordered Sequence	440	Object_History_Memory_HBB_Box.json
L3	CCME	Class-Level Change Magnitude Estimation	Image	Single Choice	615	Change_Speed_Substantial.json
L3	CSOCO	Cross-Sequence Object Construction Ordering	Geo Coordinate	Single Choice	206	Cross_View_Object_Development_Comparison_Coordinate.json
L3	CSOCO	Cross-Sequence Object Construction Ordering	HBB Box	Single Choice	204	Cross_View_Object_Development_Comparison_HBB_Box.json
L3	OCO	Object Construction Ordering	Geo Coordinate	Single Choice	483	Object_Development_Comparison_Coordinate.json
L3	OCO	Object Construction Ordering	HBB Box	Single Choice	483	Object_Development_Comparison_HBB_Box.json
L3	RDC	Region Development Comparison	HBB Box	Single Choice	177	Region_Comparison_HBB_Box.json

Competency levels — the by_category keys of metrics.json map to the results columns as:

by_category key	Level	Competency	Results column
Perception	L0	Land Cover Perception	LCP
Relation_Assessment	L1	Temporal Recognition	TR
Memory	L2	Long-Term Memory	LTM
Spatio_Temporal_Reasoning	L3	Spatio-Temporal Reasoning	STR

Level shares of the benchmark: L0 7.5%, L1 54.2%, L2 26.0%, L3 12.3% (17,689 QA pairs in total).

📁 Repository Structure

GeoChrono/
├── data_construction/       # ChronoBench / ChronoInstruct construction code
│   ├── perception/          #   L0 tasks (rule-based QA generation from change masks)
│   ├── relation_assessment/ #   L1 tasks
│   ├── memory/              #   L2 tasks
│   ├── Spatio-TemporalReasoning/  # L3 tasks
│   ├── train_set_summary/   #   ChronoInstruct assembly (3 answer formats → SFT JSON)
│   └── README.md            #   Full construction logic documentation
├── test/                    # ChronoBench evaluation harness
│   ├── run_eval.sh          #   Main entry point
│   ├── evaluate.py          #   Evaluation loop
│   ├── config.py            #   Data paths & model registry
│   ├── metrics.py           #   Single-choice / multiple-choice / ordered-sequence metrics
│   └── models/              #   Model wrappers (Qwen3-VL, InternVL3.5, EarthDial, ...)
├── lmms_eval_tasks/         # ChronoBench as an lmms-eval task pack (optional, self-contained)
│   ├── chronobench/         #   19 task + 5 group YAMLs, utils.py, verbatim system prompts
│   └── tests/               #   Parity tests against the test/ harness (no GPU needed)
├── ottc-novel/              # GeoChrono core: TempEnc training & evaluation
│   ├── ottc_module.py       #   TempEnc implementation ("OTTC" is its internal codename)
│   ├── ottc_wrapper.py      #   Injection into the Qwen3-VL forward pass
│   ├── train/  test/  scripts/
├── coarse-to-fine/          # GeoChrono + C2FComp: token compression variant
│   ├── c2f_module.py        #   C2FComp implementation
│   ├── saliency_module.py   #   Prompt-guided saliency scoring
│   └── train/  test/  scripts/
├── qwenvl/                  # Qwen3-VL fine-tuning framework (data pipeline & trainer)
├── scripts/zero2.json       # DeepSpeed ZeRO-2 config
└── requirements.txt

Naming note: throughout the code, ottc is the internal codename of TempEnc and c2f of C2FComp.

⚙️ Getting Started

1. Installation

conda create -n geochrono python=3.12 -y
conda activate geochrono

# PyTorch 2.8.0 + CUDA 12.8
pip install torch==2.8.0 torchvision==0.23.0 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu128

pip install -r requirements.txt

FlashAttention 2 (flash_attn==2.8.3) is required for both training and evaluation. If the pip build fails, install a prebuilt wheel matching your CUDA/torch version from the official releases.

2. Data preparation

Download the annotations from 🤗 Davidup1/GeoChrono-Data and the imagery from DVL-Suite:

hf download Davidup1/GeoChrono-Data --repo-type dataset --local-dir /path/to/ChronoBench-data

Then place the DVL-Suite test-split images under ChronoBench/images/ and the train-split images under ChronoInstruct/images/, following the Image Preparation section of the dataset card. The expected layout is:

ChronoBench-data/
├── ChronoBench/
│   ├── images/<city>/<tile_id>/*.tif      # DVL-Suite test split (3,469 images)
│   └── *.json                             # 19 benchmark task files
└── ChronoInstruct/
    ├── images/<city>/<tile_id>/*.tif      # DVL-Suite train split
    ├── *_train.json                       # + text / short variants
    └── qwen3vl_mixed_train_v2.json        # merged SFT file (ready to train)

3. Configuration

Point the evaluation harness to your data (or edit the defaults in test/config.py):

export CHRONOBENCH_DATA_ROOT=/path/to/ChronoBench-data/ChronoBench   # image root
export CHRONOBENCH_JSON_DIR=/path/to/ChronoBench-data/ChronoBench    # benchmark JSONs
export EVAL_OUTPUT_DIR=./results                                     # optional

Model checkpoint paths are registered in MODEL_CONFIGS inside test/config.py — edit model_path entries to match your local checkpoints.

📊 Evaluating on ChronoBench

Baseline MLLMs

cd test

# Evaluate a model on all 19 task files
bash run_eval.sh -m qwen3vl-8b

# Evaluate on specific tasks / quick smoke test
bash run_eval.sh -m internvl3_5-8b -t "Object_Classification_HBB_Box.json" -n 5

All images are fed at a fixed 1024×1024 resolution with year-labeled interleaved prompting (2005: <image> ...). Results (per-task accuracy + level-wise aggregation) are written to test/results/.

Each entry in metrics.json and predictions.json is annotated with the taxonomy of the paper, so results can be reported without consulting 🗂️ Task Reference by hand:

"by_category": {
  "Memory": { "level": "L2", "competency": "Long-Term Memory", "abbr": "LTM",
              "total": 4596, "accuracy": 0.681, ... }        // -> the LTM column
},
"by_task": {
  "Change_Speed_Substantial.json": { "level": "L3", "abbr": "CCME",
      "name": "Class-Level Change Magnitude Estimation",
      "modality": "Image", "format": "Single Choice", "accuracy": 0.729, ... }
}

These fields are purely additive: keys, metric definitions, and numbers are unchanged, so result files produced by earlier versions remain directly comparable.

GeoChrono

Download the released weights from 🤗 Davidup1/GeoChrono (or use your own training output), then:

hf download Davidup1/GeoChrono --local-dir ./output/ottc

# Edit MODEL_PATH in the script to your Qwen3-VL-4B-Instruct checkpoint, then:
bash ottc-novel/scripts/eval_ottc.sh

The script loads the base model, applies the LoRA adapter, injects TempEnc (ottc_weights.pt) into the forward pass, and runs the full benchmark. For the C2FComp variant use coarse-to-fine/scripts/eval_c2f.sh.

🧩 Evaluating with lmms-eval

ChronoBench is also packaged as an lmms-eval task, so it can be evaluated on any backend that framework supports — including API models — without going through the harness above. The task pack lives in this repository and is loaded with --include_path; contributing it upstream is planned.

# Install lmms-eval from source. The PyPI wheel omits the shared task templates
# that 811 of its own task files include, so `pip install lmms-eval` cannot build
# its task index. The [all] extra pulls decord, which the qwen2_5_vl backend needs.
git clone https://github.com/EvolvingLMMs-Lab/lmms-eval.git
pip install -e "./lmms-eval[all]"

export CHRONOBENCH_IMAGE_ROOT=/path/to/ChronoBench   # the DVL-Suite test-split images

python -m lmms_eval --model qwen3_vl --model_args pretrained=Qwen/Qwen3-VL-8B-Instruct \
  --tasks chronobench --batch_size 1 --log_samples --output_path ./logs \
  --include_path ./lmms_eval_tasks/chronobench

Tested against lmms-eval 0.7.2. --tasks chronobench runs all 19 sub-tasks; chronobench_l0…chronobench_l3 run one competency level, and each sub-task is addressable by its paper abbreviation (chronobench_ohm_hbb, chronobench_ccme, …). Annotations are pulled from the Hub automatically; only the imagery has to be prepared by hand, because it belongs to DVL-Suite and is not redistributed with the benchmark. On a cluster, also set LMMS_EVAL_DATASETS_CACHE — see the task pack README.

This pack evaluates other models on ChronoBench. It cannot run GeoChrono itself — TempEnc and C2FComp patch the Qwen3-VL forward pass and need cross-region frame grouping passed out of band, which lmms-eval has no way to express. Reproduce GeoChrono with ottc-novel/scripts/eval_ottc.sh as described above.

The answer parser is a verbatim port of test/metrics.py and the prompt carries the same content in the same order, both locked down by parity tests that need no GPU. The one thing that varies is where the task instructions sit. By default the pack injects the per-task system prompt at the start of the user prompt, ahead of the year-labelled frames, instead of emitting a system turn — so whatever system message a backend adds cannot displace it:

# Default — works on every backend, qwen3_vl included. Nothing to set.
python -m lmms_eval --model qwen3_vl --model_args pretrained=Qwen/Qwen3-VL-8B-Instruct \
  --tasks chronobench --include_path ./lmms_eval_tasks/chronobench

# CHRONOBENCH_SYSTEM_TURN=1 — keeps the instructions in a real system turn, which
# makes the prompt byte-identical to the reference harness. Only on backends that
# pass a task's messages through untouched (qwen2_5_vl, internvl_hf).
CHRONOBENCH_SYSTEM_TURN=1 python -m lmms_eval --model qwen2_5_vl \
  --model_args pretrained=Qwen/Qwen2.5-VL-7B-Instruct \
  --tasks chronobench --include_path ./lmms_eval_tasks/chronobench

On qwen3_vl, please leave the variable unset: that backend prepends a system message of its own, and Qwen's chat template keeps only the first — so the task's own system turn would be dropped, answer-format instruction included. Prepending the instructions to the user prompt is what sidesteps that. It is an upstream limitation affecting any task that supplies its own system turn; rather than ask anyone to patch an installed dependency, the pack works around it, and we intend to propose a fix to lmms-eval.

For Qwen3-VL we recommend the test/ pipeline, which is what the paper's numbers come from; running it through lmms-eval is fine too, on the default settings. Under that default a four-task smoke test with Qwen3-VL-4B-Instruct scored 78/80 samples identically, within ±0.05 per task. Say which path produced a number when publishing. See lmms_eval_tasks/chronobench/README.md for details.

# Prompt parity: rebuilds all 17,689 prompts both ways and diffs them
CHRONOBENCH_JSON_DIR=/path/to/ChronoBench python lmms_eval_tasks/tests/test_prompt_parity.py

# Metric parity: replays your own past predictions.json through both scorers
CHRONOBENCH_RESULTS_DIR=./test/results python lmms_eval_tasks/tests/test_parity.py

The test/ workflow is unaffected. lmms-eval is an additional entry point, not a replacement — nothing in test/ changed, and results produced with it remain valid and comparable. See lmms_eval_tasks/chronobench/README.md for the full task list and metric definitions.

🚀 Training

GeoChrono is trained in a single stage on ChronoInstruct: the vision encoder and projector are frozen, TempEnc is randomly initialized and fully fine-tuned, and the LLM backbone is tuned with LoRA (r=32, α=64). Edit the paths at the top of the script, then launch from the repository root:

# GeoChrono (TempEnc)
# Set MODEL_PATH / DATA_PATH / IMAGE_FOLDER first:
#   MODEL_PATH   → Qwen3-VL-4B-Instruct checkpoint
#   DATA_PATH    → ChronoInstruct/qwen3vl_mixed_train_v2.json
#   IMAGE_FOLDER → ChronoInstruct/ (directory containing images/)
NPROC_PER_NODE=4 bash ottc-novel/scripts/train_ottc.sh

# GeoChrono + C2FComp
NPROC_PER_NODE=4 bash coarse-to-fine/scripts/train_c2f.sh

Training uses DeepSpeed ZeRO-2 (scripts/zero2.json), bf16, one epoch, LR 1e-4 for LoRA and 5e-4 for TempEnc, and --max_pixels 1048576 (1024×1024 input). The reference setup is 4× H100 80GB.

🔨 Building the Data from Scratch

The complete rule-based construction pipeline is provided in data_construction/. It requires a few extra geospatial dependencies on top of the base environment — most notably rasterio for pixel ↔ geographic coordinate projection over the NAIP GeoTIFFs:

pip install -r data_construction/requirements.txt

Pipeline components:

• Per-level generation scripts consume DVL-Suite semantic change masks and emit task JSONs (see data_construction/README.md for the mask encoding rules and per-task logic).
• train_set_summary/convert_to_llava_v2.py assembles the three answer-format variants into the final SFT file.
• Appendix docs cover the benchmark construction, prompt templates, and evaluation setup.

Note: several construction steps involve unseeded random sampling, so regenerated JSONs will not be byte-identical to the released ones. For reproducible comparison, evaluate on the released ChronoBench JSONs.

📈 Results

Method	LCP	TR	LTM	STR	OA
Human	97.04	89.78	91.73	95.56	92.28
Gemini-3-Flash	65.51	61.38	47.52	59.89	57.48
GPT-5.4	43.53	67.57	39.21	50.42	56.29
Qwen3-VL-32B	43.76	57.88	24.06	47.23	46.73
DVLChat-4B	47.74	34.21	22.91	40.59	44.07
GeoChrono (ours)	88.65	83.03	68.10	72.92	78.34

(LCP: Land Cover Perception, TR: Temporal Recognition, LTM: Long-Term Memory, STR: Spatio-Temporal Reasoning, OA: Overall Accuracy. Full comparison across 19 MLLMs in the paper.)

📚 Citation

If you find ChronoBench, ChronoInstruct, or GeoChrono useful, please cite our paper:

@article{li2026geochrono,
  title   = {GeoChrono: Benchmarking and Rethinking Long-Term Temporal Understanding in Remote Sensing},
  author  = {Li, Yujie and Pan, Jiancheng and Wei, Zhiwei and Wang, Jiuniu and Peng, Mugen and Xu, Wenjia},
  journal = {arXiv preprint arXiv:2607.15768},
  year    = {2026}
}

📖 References and Acknowledgements

• Code: our fine-tuning framework builds on the official Qwen3-VL fine-tuning codebase; the base model is Qwen3-VL-4B-Instruct.
• Data: ChronoBench and ChronoInstruct build on the human-annotated semantic change masks of DVL-Suite / DynamicVL; imagery originates from the USDA NAIP program.

The code in this repository, the ChronoBench / ChronoInstruct data, and the GeoChrono weights are all released under the Apache License 2.0 (see LICENSE). The underlying DVL-Suite imagery and annotations remain subject to their own license terms.

📮 Contact

For questions or feedback, please open an issue or contact liyujie2003@bupt.edu.cn.