Egocentric Action Anticipation by Disentangling Encoding and Inference