Has anyone else noticed extreme slowness when moving from PyTorch 1.0 to 1.1? I moved my pyro code using SVI from PyTorch 1.0 to 1.1 and found astonishing delays. I’m using the dev branch of pyro-ppl, and I’m using the google “deep learning” pytorch containers (so they get built by someone who knows what they’re doing there).
The worst instance of delays in in the JIT compiling. The JIT compile step is almost unbearably slow now. The compile step took almost 180 seconds on pytorch 1.0. It takes over 1000 seconds in 1.1. Disabling optimization during JIT provides little relief. SVI steps subsequent to the JIT step are also slower in 1.1 - about twice as slow as 1.0.
I’m also seeing about a 17% slowdown from 1.0 to 1.1 without JIT compiling.
Has anyone else seen this slowdown? I presume the google folks built their pytorch containers the same way, so hopefully the pytorch build isn’t the issue… I can roll back to 1.0 for now, but I’m wondering if something structural didn’t change that might need to be addressed. I’m guessing this isn’t limited to me??
Here are times from the 1.0 run:
print(F"torch version {torch.__version__}") torch version 1.0.1.post2 reset_params() elbo = TraceEnum_ELBO(max_plate_nesting=1) svi = SVI(model.model_fn, model.guide, optim, elbo) with Timer('first step'): svi.step(yseq, xseq, length=len(yseq), **model_kwargs) with Timer('100 more steps'): for i in range(100): svi.step(yseq, xseq, length=len(yseq), **model_kwargs) [first step] Elapsed: 2.387s [100 more steps] Elapsed: 232.172s reset_params() elbo = JitTraceEnum_ELBO(max_plate_nesting=1, jit_options={"optimize": True}) svi = SVI(model.model_fn, model.guide, optim, elbo) with Timer('first step'): svi.step(yseq, xseq, length=len(yseq), **model_kwargs) with Timer('100 more steps'): for i in range(100): svi.step(yseq, xseq, length=len(yseq), **model_kwargs) [first step] Elapsed: 176.492s [100 more steps] Elapsed: 83.862s reset_params() elbo = JitTraceEnum_ELBO(max_plate_nesting=1, jit_options={"optimize": False}) svi = SVI(model.model_fn, model.guide, optim, elbo) with Timer('first step'): svi.step(yseq, xseq, length=len(yseq), **model_kwargs) with Timer('100 more steps'): for i in range(100): svi.step(yseq, xseq, length=len(yseq), **model_kwargs) [first step] Elapsed: 164.959s [100 more steps] Elapsed: 163.283s
And here are the same times from the 1.1 run:
print(F"torch version {torch.__version__}") torch version 1.1.0 reset_params() elbo = TraceEnum_ELBO(max_plate_nesting=1) svi = SVI(model.model_fn, model.guide, optim, elbo) with Timer('first step'): svi.step(yseq, xseq, length=len(yseq), **model_kwargs) with Timer('100 more steps'): for i in range(100): svi.step(yseq, xseq, length=len(yseq), **model_kwargs) [first step] Elapsed: 3.982s [100 more steps] Elapsed: 271.525s reset_params() elbo = JitTraceEnum_ELBO(max_plate_nesting=1, jit_options={"optimize": True}) svi = SVI(model.model_fn, model.guide, optim, elbo) with Timer('first step'): svi.step(yseq, xseq, length=len(yseq), **model_kwargs) with Timer('100 more steps'): for i in range(100): svi.step(yseq, xseq, length=len(yseq), **model_kwargs) [first step] Elapsed: 1013.402s [100 more steps] Elapsed: 190.331s reset_params() elbo = JitTraceEnum_ELBO(max_plate_nesting=1, jit_options={"optimize": False}) svi = SVI(model.model_fn, model.guide, optim, elbo) with Timer('first step'): svi.step(yseq, xseq, length=len(yseq), **model_kwargs) with Timer('100 more steps'): for i in range(100): svi.step(yseq, xseq, length=len(yseq), **model_kwargs) [first step] Elapsed: 1041.228s [100 more steps] Elapsed: 185.235s