How to get progression in Multi-GPUs mode

campagne · August 13, 2021, 9:39am

Hello,
I’m using 2 GPUs to run NUTS (2 chains), and get the following message

numpyro/util.py:313: UserWarning: We will disable progress bar because it does not work yet on multi-GPUs platforms.
“We will disable progress bar because it does not work yet on multi-GPUs platforms.”

Is there a workaround to get some information on the progression of the 2 chains? Thanks

fehiepsi · August 13, 2021, 12:51pm

Currently, it is blocked by this upstream issue. Could you double check if it has been resolved. I don’t have 2 GPUs to test.

campagne · August 13, 2021, 1:07pm

Heuuu. what I should double check?

fehiepsi · August 13, 2021, 3:13pm

How about switching this line from False to True?

campagne · August 13, 2021, 3:31pm

Ha I see, I let u know asap. Thanks

Well it seems that the program is idle

But the job is still alive and run

but the progress bar of chain 1 is not updated and for chain 0 the refreshing frequency is very low. And there is not number of steps/step size information

Well it is strange now the time spent for the two chains…