Are You Forgetting the Future and Predicting the Past?