Has anyone else noticed extreme slowness when moving from PyTorch 1.0 to 1.1? I moved my pyro code using SVI from PyTorch 1.0 to 1.1 and found astonishing delays. I’m using the dev branch of pyro-ppl, and I’m using the google “deep learning” pytorch containers (so they get built by someone who knows what they’re doing there).
The worst instance of delays in in the JIT compiling. The JIT compile step is almost unbearably slow now. The compile step took almost 180 seconds on pytorch 1.0. It takes over 1000 seconds in 1.1. Disabling optimization during JIT provides little relief. SVI steps subsequent to the JIT step are also slower in 1.1 - about twice as slow as 1.0.
I’m also seeing about a 17% slowdown from 1.0 to 1.1 without JIT compiling.
Has anyone else seen this slowdown? I presume the google folks built their pytorch containers the same way, so hopefully the pytorch build isn’t the issue… I can roll back to 1.0 for now, but I’m wondering if something structural didn’t change that might need to be addressed. I’m guessing this isn’t limited to me??
Here are times from the 1.0 run:
print(F"torch version {torch.__version__}") torch version 1.0.1.post2 reset_params() elbo = TraceEnum_ELBO(max_plate_nesting=1) svi = SVI(model.model_fn, model.guide, optim, elbo) with Timer('first step'): svi.step(yseq, xseq, length=len(yseq), **model_kwargs) with Timer('100 more steps'): for i in range(100): svi.step(yseq, xseq, length=len(yseq), **model_kwargs) [first step] Elapsed: 2.387s [100 more steps] Elapsed: 232.172s reset_params() elbo = JitTraceEnum_ELBO(max_plate_nesting=1, jit_options={"optimize": True}) svi = SVI(model.model_fn, model.guide, optim, elbo) with Timer('first step'): svi.step(yseq, xseq, length=len(yseq), **model_kwargs) with Timer('100 more steps'): for i in range(100): svi.step(yseq, xseq, length=len(yseq), **model_kwargs) [first step] Elapsed: 176.492s [100 more steps] Elapsed: 83.862s reset_params() elbo = JitTraceEnum_ELBO(max_plate_nesting=1, jit_options={"optimize": False}) svi = SVI(model.model_fn, model.guide, optim, elbo) with Timer('first step'): svi.step(yseq, xseq, length=len(yseq), **model_kwargs) with Timer('100 more steps'): for i in range(100): svi.step(yseq, xseq, length=len(yseq), **model_kwargs) [first step] Elapsed: 164.959s [100 more steps] Elapsed: 163.283s
And here are the same times from the 1.1 run:
print(F"torch version {torch.__version__}") torch version 1.1.0 reset_params() elbo = TraceEnum_ELBO(max_plate_nesting=1) svi = SVI(model.model_fn, model.guide, optim, elbo) with Timer('first step'): svi.step(yseq, xseq, length=len(yseq), **model_kwargs) with Timer('100 more steps'): for i in range(100): svi.step(yseq, xseq, length=len(yseq), **model_kwargs) [first step] Elapsed: 3.982s [100 more steps] Elapsed: 271.525s reset_params() elbo = JitTraceEnum_ELBO(max_plate_nesting=1, jit_options={"optimize": True}) svi = SVI(model.model_fn, model.guide, optim, elbo) with Timer('first step'): svi.step(yseq, xseq, length=len(yseq), **model_kwargs) with Timer('100 more steps'): for i in range(100): svi.step(yseq, xseq, length=len(yseq), **model_kwargs) [first step] Elapsed: 1013.402s [100 more steps] Elapsed: 190.331s reset_params() elbo = JitTraceEnum_ELBO(max_plate_nesting=1, jit_options={"optimize": False}) svi = SVI(model.model_fn, model.guide, optim, elbo) with Timer('first step'): svi.step(yseq, xseq, length=len(yseq), **model_kwargs) with Timer('100 more steps'): for i in range(100): svi.step(yseq, xseq, length=len(yseq), **model_kwargs) [first step] Elapsed: 1041.228s [100 more steps] Elapsed: 185.235s
If you were to JIT your model/guide code directly (i.e. outside the context of inference), do you still notice the slowdown?
I use JIT for HMC in Pyro almost all the time, because it easily leads to a 4x difference in performance, but then HMC falls precisely in that sweet spot where some additional time spent in model compilation and first run is easily amortized over many thousands of forward / backward passes through the model.