run so slow

[shangfenghuang]

Thank your work. When I run the work，the speed of running one epoch is very slow. it is about one hour. But I see in the log file that the running one epoch just 20 minutes. So I can't understand. Can you help me?
this is the log file in one epoch
2020-12-03 20:29:16.650905
---- batch: 050 ----
mean loss: 184.534502
accuracy: 0.414029
---- batch: 100 ----
mean loss: 135.335496
accuracy: 0.530120
---- batch: 150 ----
mean loss: 129.747210
accuracy: 0.535523
---- batch: 200 ----
mean loss: 122.139866
accuracy: 0.558028
---- batch: 250 ----
mean loss: 119.617929
accuracy: 0.560829
---- batch: 300 ----
mean loss: 113.885079
accuracy: 0.584236
---- batch: 350 ----
mean loss: 116.340719
accuracy: 0.569333
---- batch: 400 ----
mean loss: 113.555371
accuracy: 0.580139
---- batch: 450 ----
mean loss: 110.236181
accuracy: 0.584645
---- batch: 500 ----
mean loss: 110.504616
accuracy: 0.594045
---- batch: 550 ----
mean loss: 104.735284
accuracy: 0.608489
---- batch: 600 ----
mean loss: 104.923058
accuracy: 0.601770
---- batch: 650 ----
mean loss: 106.618274
accuracy: 0.598362
---- batch: 700 ----
mean loss: 104.379048
accuracy: 0.612791
---- batch: 750 ----
mean loss: 102.553243
accuracy: 0.618261
---- batch: 800 ----
mean loss: 102.361754
accuracy: 0.611996
---- batch: 850 ----
mean loss: 102.780255
accuracy: 0.615874
---- batch: 900 ----
mean loss: 100.150999
accuracy: 0.626256
---- batch: 950 ----
mean loss: 104.170692
accuracy: 0.608262
---- batch: 1000 ----
mean loss: 102.701527
accuracy: 0.615849
---- batch: 1050 ----
mean loss: 101.427789
accuracy: 0.613831
---- batch: 1100 ----
mean loss: 104.226453
accuracy: 0.599298
---- batch: 1150 ----
mean loss: 97.109982
accuracy: 0.631632
---- batch: 1200 ----
mean loss: 99.082409
accuracy: 0.623211
---- batch: 1250 ----
mean loss: 98.161291
accuracy: 0.618052
---- batch: 1300 ----
mean loss: 93.044155
accuracy: 0.639523
---- batch: 1350 ----
mean loss: 90.239651
accuracy: 0.652435
---- batch: 1400 ----
mean loss: 90.905718
accuracy: 0.650580
---- batch: 1450 ----
mean loss: 90.796373
accuracy: 0.650573
---- batch: 1500 ----
mean loss: 88.180042
accuracy: 0.663508
---- batch: 1550 ----
mean loss: 90.931050
accuracy: 0.647735
---- batch: 1600 ----
mean loss: 90.572594
accuracy: 0.647632
---- batch: 1650 ----
mean loss: 83.347111
accuracy: 0.676880
---- batch: 1700 ----
mean loss: 88.313284
accuracy: 0.657798
---- batch: 1750 ----
mean loss: 82.861588
accuracy: 0.681264
---- batch: 1800 ----
mean loss: 89.783586
accuracy: 0.651877
---- batch: 1850 ----
mean loss: 84.404577
accuracy: 0.673862
---- batch: 1900 ----
mean loss: 87.348818
accuracy: 0.658631
---- batch: 1950 ----
mean loss: 83.427303
accuracy: 0.670092
---- batch: 2000 ----
mean loss: 88.491244
accuracy: 0.654787
---- batch: 2050 ----
mean loss: 84.942625
accuracy: 0.661988
---- batch: 2100 ----
mean loss: 84.637836
accuracy: 0.667242
---- batch: 2150 ----
mean loss: 86.843850
accuracy: 0.660539
---- batch: 2200 ----
mean loss: 85.992690
accuracy: 0.670484
---- batch: 2250 ----
mean loss: 86.092916
accuracy: 0.659830
---- batch: 2300 ----
mean loss: 82.865510
accuracy: 0.679619
---- batch: 2350 ----
mean loss: 82.640754
accuracy: 0.674528
---- batch: 2400 ----
mean loss: 81.347898
accuracy: 0.683257
---- batch: 2450 ----
mean loss: 83.726160
accuracy: 0.670507
---- batch: 2500 ----
mean loss: 82.711281
accuracy: 0.667460
---- batch: 2550 ----
mean loss: 85.248889
accuracy: 0.664610
---- batch: 2600 ----
mean loss: 79.271644
accuracy: 0.684864
---- batch: 2650 ----
mean loss: 82.488315
accuracy: 0.672837
---- batch: 2700 ----
mean loss: 81.616334
accuracy: 0.676569
---- batch: 2750 ----
mean loss: 83.177334
accuracy: 0.668547
---- batch: 2800 ----
mean loss: 81.139334
accuracy: 0.684465
---- batch: 2850 ----
mean loss: 80.436449
accuracy: 0.679211
---- batch: 2900 ----
mean loss: 80.295713
accuracy: 0.678259
---- batch: 2950 ----
mean loss: 80.749244
accuracy: 0.671857
---- batch: 3000 ----
mean loss: 80.518642
accuracy: 0.677207
---- batch: 3050 ----
mean loss: 77.829687
accuracy: 0.685728
---- batch: 3100 ----
mean loss: 81.392671
accuracy: 0.671245
---- batch: 3150 ----
mean loss: 76.950525
accuracy: 0.691033
---- batch: 3200 ----
mean loss: 79.833296
accuracy: 0.682424
---- batch: 3250 ----
mean loss: 81.639724
accuracy: 0.670625
---- batch: 3300 ----
mean loss: 77.314783
accuracy: 0.688428
---- batch: 3350 ----
mean loss: 76.034729
accuracy: 0.694535
---- batch: 3400 ----
mean loss: 78.178265
accuracy: 0.684664
---- batch: 3450 ----
mean loss: 75.660341
accuracy: 0.692333
---- batch: 3500 ----
mean loss: 74.944008
accuracy: 0.687972
---- batch: 3550 ----
mean loss: 77.615459
accuracy: 0.687553
---- batch: 3600 ----
mean loss: 77.393342
accuracy: 0.685459
---- batch: 3650 ----
mean loss: 80.323210
accuracy: 0.676606
---- batch: 3700 ----
mean loss: 77.831140
accuracy: 0.678844
---- batch: 3750 ----
mean loss: 73.645795
accuracy: 0.701312
---- batch: 3800 ----
mean loss: 73.109120
accuracy: 0.698930
---- batch: 3850 ----
mean loss: 72.719140
accuracy: 0.707183
---- batch: 3900 ----
mean loss: 76.973215
accuracy: 0.686412
---- batch: 3950 ----
mean loss: 72.995662
accuracy: 0.698651
---- batch: 4000 ----
mean loss: 74.334438
accuracy: 0.692604
---- batch: 4050 ----
mean loss: 71.758526
accuracy: 0.710445
---- batch: 4100 ----
mean loss: 73.972742
accuracy: 0.695303
---- batch: 4150 ----
mean loss: 70.600237
accuracy: 0.705352
---- batch: 4200 ----
mean loss: 71.107945
accuracy: 0.703613
training one batch require 791.24 milliseconds
2020-12-03 21:49:02.473758
---- EPOCH 000 EVALUATION ----
eval mean loss: 12.938971
eval overall accuracy: 0.732570
eval avg class acc: 0.566511
eval mIoU of other20: 0.432905
eval mIoU of wall: 0.602921
eval mIoU of floor: 0.916897
eval mIoU of cabinet: 0.323481
eval mIoU of bed: 0.547132
eval mIoU of chair: 0.734238
eval mIoU of sofa: 0.625162
eval mIoU of table: 0.548581
eval mIoU of door: 0.265242
eval mIoU of window: 0.233336
eval mIoU of bookshelf: 0.480115
eval mIoU of picture: 0.001725
eval mIoU of counter: 0.326607
eval mIoU of desk: 0.312228
eval mIoU of curtain: 0.325191
eval mIoU of refridgerator: 0.171487
eval mIoU of shower curtain: 0.188239
eval mIoU of toilet: 0.371836
eval mIoU of sink: 0.325391
eval mIoU of bathtub: 0.414665
eval mIoU of otherfurniture: 0.162511
eval mIoU of all classes: 0.395709
testing one batch require 334.10 milliseconds
Model saved in file: /home/disk1/hsf/SPH3D-GCN/log_scannet/model.ckpt-0

[shangfenghuang]

my GPU just is used about 4000 memory.

[EnyaHermite]

You can use less buffer_size in the input_fn. Shuffling tfrecord dataset with 10000 buffer_size setting indeed burdens the CPU memory. Reduce it to 1000 should also work well.