Gradient accumulation with distributed training

[IvanUkhov]

Does anybody know if the implemented gradient accumulation can run under tf.distribute.MirroredStrategy?

[IvanUkhov]

Opened an issue: #20582.

[abhaskumarsinha]

If you are using tf backend, you can use gradient accumulation old TF way with it.

[IvanUkhov]

What is this old way?

[roebel]

I wanted to try this as well, but the problem currently appears to be that MirroredStrategy is broken in the latest versions of keras. see #21061

Besides that, I don't think there is an official old way. That's why there were a few wrappers in the wild as for example this one here: https://gradientaccumulator.readthedocs.io/en/latest/background/gradient_accumulation.html

[innat]

keras-team/tf-keras#301