examples/tutorial/prithvi_v2_eo_300_tl_unet_burnscars.ipynb

[Atmoboran]

Hello,

thanks for the new tutorials. This is exactly what I was looking for! It makes the whole process of fine-tuning much easier to understand.
Unfortunately I think it doesn't correctly work for me.

So with in the default notebook (I haven't changed anything):
cell 14: best_ckpt_path = "output/burnscars/checkpoints/best-epoch=01.ckpt"
-> This file does not exist, instead only 'best-epoch=00.ckpt' exists, but I guess those are the weights at inialization (?), which will predict 'no burned' everywhere.

So I tried training with more (15) epochs and not freeze the encoder, which also did not produce any more checkpoint files.

Then I included 'save_top_k=5, save_last=True' into the callback function, which produces multiple 'best-epoch=XX.ckpt' files. I used a newer .ckpt file, i.e. 'best-epoch=04.ckpt', but it also only predicts 'no burned'.

Something is going wrong, especially as the Loss is NaN. These are my metrics for the third test:

[{'test/loss': nan,
'test/Multiclass_Accuracy': 0.8957206606864929,
'test/multiclassaccuracy_no burned': 1.0,
'test/multiclassaccuracy_burned': 0.0,
'test/Multiclass_F1_Score': 0.8957206606864929,
'test/Multiclass_Jaccard_Index': 0.44786033034324646,
'test/multiclassjaccardindex_no burned': 0.8957206606864929,
'test/multiclassjaccardindex_burned': 0.0,
'test/Multiclass_Jaccard_Index_Micro': 0.8111359477043152}]

I'm training on NVIDIA RTXA6000-48Q with CUDA Version: 12.2

Best regards,
Atmoboran

[Joao-L-S-Almeida]

Hi, @Atmoboran. Just to inform you, I'm checking and trying to reproduce your issue.
Thanks.

[Atmoboran]

Hey, how is it going? Were you able to reproduce the issue or does it work for you?

[Atmoboran]

Hi @Joao-L-S-Almeida,

here I have my ipynb file attached. So if that helps you can look at my outputs.

prithvi_v2_eo_300_tl_unet_burnscars.zip

Thank you!

[Joao-L-S-Almeida]

@Atmoboran The notebook is failing for me, but for others reasons. I'll run it with a different environment:

File [~/.pyenv/versions/3.12.7/lib/python3.12/inspect.py:3273](http://localhost:8889/home/jalmeida/.pyenv/versions/3.12.7/lib/python3.12/inspect.py#line=3272), in Signature.bind(self, *args, **kwargs)
   3268 def bind(self, [/](http://localhost:8889/), *args, **kwargs):
   3269     """Get a BoundArguments object, that maps the passed `args`
   3270     and `kwargs` to the function's signature.  Raises `TypeError`
   3271     if the passed arguments can not be bound.
   3272     """
-> 3273     return self._bind(args, kwargs)

RecursionError: maximum recursion depth exceeded

[Atmoboran]

So I have found the problem while watching @paolo-fraccaro 's workshop on youtube (https://www.youtube.com/watch?v=CB3FKtmuPI8).
There are quite a few mismatches between the notebook he is showing and the one that can be downloaded from terratorch/examples/tutorial/prithvi_v2_eo_300_tl_unet_burnscars.ipynb

The most important mismatch and the reason why I was getting NaN as a loss is inside:
datamodule = terratorch.datamodules.GenericNonGeoSegmentationDataModule(
...
no_data_replace = 0,
no_label_replace = -1,
)

If this is included, then the script works!

[Atmoboran]

The same thing is missing in: prithvi_v2_eo_300_tl_unet_sen1floods11.ipynb

[Atmoboran]

The other mismatches are (don't thinkt they are dramatic, but just so you know):

batch_size=4,
"backbone": "prithvi_eo_v2_300" # no _tl
"backbone_img_size": 512
loss="ce"

[paolofraccaro]

Hi @Atmoboran ! The ones I showed are actually here: https://github.com/IBM/terratorch/tree/organise_examples/examples/notebooks/Prithvi_EO

We are in the process of reorg the examples and we did not manage to transition them yet!

Sorry about it and thanks for reaching out!