Question about using model on my datasets

[keruoya]

"Hello! Thank you for your fantastic work. I'm training a model with my dataset, but the fake images from the 20th epoch are completely gray. Could this be due to an error in my operations, or because I haven't pre-trained the generator and I'm not using the models you provided?"

[keruoya]

And this is my history.csv
gen_ab,gen_ba,cycle_a,cycle_b,disc_a,disc_b,idt_a,idt_b,gp_a,gp_b,epoch,time
0.41484783540169395,0.3107682885809077,0.34295186284515594,0.3679717449678315,0.22416516832179492,0.1776486219631301,0.06967297034958998,0.16064514741301536,0.005031600462583204,0.013559102652490967,20

[usert5432]

Hi @keruoya,

The history file looks ok to me. The resulting image also contains some information (I played with color thresholds in an image editor and there is a clear hidden structure in the image).

I think, it is possible that the model was not trained long enough (with epoch = 20, this may happen if the training dataset is very small)? For reference, our generator is exponentially averaged with momentum of 0.9999. So, it takes ~10000 training iterations to just overwrite the initial (random) state of the generator.

I was wondering if it would be possible to continue the training for a little bit longer and see if it get better?

Otherwise, there are are few other things one may check to investigate this issue:
a. CycleGAN translates images in both directions. Could you also check the corresponding image in the opposite direction? Is it also gray?
b. CycleGAN outputs cyclically-translated images, which are saved under reco_a and reco_b subdirectories. Could you check if those images match real_a and real_b images? And that real_a and real_b images look as expected (if not, there may be a problem with dataloaders)?

[keruoya]

Thank you for your suggestion! I will continue with the training. I also checked the fake_b, reco_a, reco_b of the 20th epoch, they are all gray like fake_a, which may also contain hidden information? Then real_a and real_b are normal, so there shouldn't be any issue with the dataloader.And here is fake_a,fake_b,reco_a,reco_b.

[keruoya]

And the size of my datasets is 4000,is this too small?Thank you for your suggestion!

[usert5432]

Hi @keruoya,

Thank you for elaboration!

And the size of my datasets is 4000,is this too small?

No, this should be more than enough (assuming 4000 training samples).

Looking over your images, I do see some structure in both fake_a and fake_b, but no structure in reco_a and reco_b. In my experience, when the generators fail to recover reco_a and reco_b -- it usually helps to increase lambda_a and lambda_b parameters of model.

So, I would recommend increasing these values by a factor of 10 (or 100, depending on the dataset) and trying the training again. If that still does not help, another thing that may be helpful is replacing normal weight initialization of the generators by the kaiming one.

[Squishedmac]

hey just wanted to know if any of the above mentioned steps helped to produce a colour accurate image and not just a gray one, i have the same issue and tweaking lambda_a and lambda_b just made hte gen_ab and gen_ba loss explode up

[keruoya]

hi, @Squishedmac I just replace weight initialization of the generators by the one.normal kaiming and continue training,and here is my loss