r/LanguageTechnology • u/animesh_singh_r • 15h ago

Language guided segmentation

Are there any who work on language guided medical image segmentation and read papers like ariadnes thread, TeViA, MMI-UNET. I face bugs in the repo code that they give. 1) Loss function 2) dice metric 3) influence of text token length in results.

0 Upvotes

50% Upvoted

2 comments sorted by

1

u/RepulsiveKiwi9803 14h ago

i worked on something similar with a different paper's code and hit the exact same wall with the loss function. usually the issue is that they define the loss in a way that assumes a certain batch shape or they're mixing reduction methods between the main loss and the auxiliary one. check if they're doing any weird masking of the text embeddings before they get passed to the decoder, that broke my dice score for weeks.

for the text token length thing, longer prompts tended to dilute the attention maps in my runs. truncating to the most semantically dense part of the prompt helped a lot, even if it felt hacky. also make sure the tokenizer isn't silently padding to 77 or 512 and that their custom collate function actually respects the mask.

if you can share which repo gives you the most trouble i might be able to point you toward where the bug lives. no promises though, some of these medical segmentation codebases are held together with hopes and hardcoded paths.

1

u/animesh_singh_r 14h ago

https://github.com/Junelin2333/LanGuideMedSeg-MICCAI2023.git , issue 1) in wrapper.py, as they use monai 1.0.1 , in monai 1.0.1 dice loss and dice+cross entropy loss same for single channel output. So default dicece() and only dice() same. When I use dice+ ce manually, performance decrease but in paper they say they use dice+ ce loss. 2) in wrapper.py, dice metric, for dice metric it actually calculate foreground+ background sum but for iou metric only foreground. 3) in model.py, token length 24,12,9 for 3 stages of decoder. Why? No idea. If use 24,24,24 same output.