test mIoU

[Linbooo]

Hello,
I have a pretty slow GPU. It will probably takes weeks for me to train all 1500 epochs :( So far I have trained 300 epochs, got test mIoU 45.6 which is far from yours (47.8), using my own test script. I want to ask if my result is reasonable, can it be near 47.8 when all epochs are done? or I have to do some other tricks to reach there such as multiscale evaluation? By the way, I did not take the unlabeled class into consideration when I calculated the test mIoU. I would appreciate it if you can share with us your test script and metric calculations :)) Thanks for the project code anyway 👍

/Linbo

[JindongJiang]

Hi Linbo,

If you are using the same batchsize as in the paper, the highest mIoU should be around 400 - 480. No other trick is required to achieve this result, but I did search the best result in the saved checkpoints.

Also, I will release the metric calculation code later.

Best,

Jindong

[Linbooo]

Thanks! have you tried using larger base learning rate? I do not have a powerful GPU cluster and the process is too slow for me :(( 感谢了

[Linbooo]

or have you tried to use 'poly' learning rate schedule instead of 'step'?

[JindongJiang]

I haven't tried these two, but you can certainly try using them.

[Linbooo]

Hi again. I have now trained 500 epochs without any stop. I tested each 5th epoch between 400 and 500 on test set, i.e. 405, 410, 415... The mIoU are always between 45.0 to 45.6. Im pretty sure that my metric is correct since it got 47.8 on the weights provided by you. The training used the exact default setting in your code. Should I let it train some more days, what do you think

[JindongJiang]

Hi, what about now? Still 45.6?

[Linbooo]

Sadly, no process at all. Now I have trained to 1100 epochs. All results after 500 epochs got worse than before. I think the highest mIoU I have now is 45.6 so far. I think I need to do it like you wrote in the paper, which is checking the validation loss as you have trained for a certain period. Then picking up that checkpoint which has lowest val loss and reduce the learning rate, and continue to train. Loop this process many times. Is this approach you have used??

[JindongJiang]

That makes sense, results after 500 are basically overfitted. And Yes, I suggest you do that also. By the way, are you using PyTorch 0.4 or 1.0? I remember that the weight initialization of PyTorch had changed from 0.4 to 1.0?

[Linbooo]

I used 0.4.1, I think this is what you've used right? I just found that you used batch-size 5 for the training, but you wrote "default=10" in help. What is the actual batch-size you used?

[JindongJiang]

Yes, that is correct. And I use 5 as the training batch size, sorry for the misleading help message.

[hxwork]

Hello，
I trained the model use pytorch 0.4.0 and the original settings. After 220 epochs I test the model use my own evaluate codes on the validation datasets , but the mIOU result can only achieve 32.0. I don't understand why the point is so low....I sincerely hope that you can release the evaluate code soon. Thanks for the project code anyway!!!

[Linbooo]

32.0 seems too low. I can provide you my evaluation code for both mIoU and Accuracy. I have tested it on the weights given by JindongJiang, and I got correct values for both metrics. Not sure if it is allowed to post code here. Can you give me your email

[hxwork]

That's owsome! my email is 851960956@qq.com. Thank you very much!!!

[hxwork]

I use your test code and the mIOU result is 42.63! My code may have some mistakes. Thank you!

[Linbooo]

nice nice! I think I got similar results as yours. As I said, the best one is 45.6 I got. Also as JindongJiang suggested, we need to do the exhaustive search in the validation loss to obtain 47.8, which I still doubt the validity, to be honest. It will take forever for my computer to train and test. If you ever reach 47.8, tell me. :))