[Question] How should frame pooling on *-v5 Atari envs be applied?

[skandermoalla]

Question

It is not clear how frame pooling (component-wise maximum over two consecutive frames [1,2]) should be applied with the *-v5 version of the Atari environments as they implement frame skipping internally.
As far as I understand pooling is generally applied to the last two environment frames (i.e. the last skipped frame and the one after that would be shown to the agent). Without access to the skipped frame, pooling would be applied to the last two frames shown to the agent and would defeat the purpose of capturing alternating sprites [1, 2].

I couldn't find any documentation on this or a relevant wrapper for it. Any clarification is welcome. Thanks!

[1] https://danieltakeshi.github.io/2016/11/25/frame-skipping-and-preprocessing-for-deep-q-networks-on-atari-2600-games/
[2] Machado et al. “Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents” Journal of Artificial Intelligence Research (2018) URL: https://jair.org/index.php/jair/article/view/11182.

[pseudo-rnd-thoughts]

Great question, for some reason I thought that this was done internally.

I found this issue discussing it - Farama-Foundation/Arcade-Learning-Environment#467

There is a open pr with a proposed solution but doesn't look it has been accepted yet

[skandermoalla]

Thanks a lot for the pointer! Should we keep this issue open to track it and then add it to the Gymnasium documentation?

[pseudo-rnd-thoughts]

Yes it is probably a good idea to note this to prevent future users from having the same issue

[skandermoalla]

@pseudo-rnd-thoughts do you have any updates that completed this issue?

[pseudo-rnd-thoughts]

You can specify a frameskip=1 with gym.make("ALE/Pong-v5", frameskip=1) which would allow you to implement the frameskip yourself like the AtariPreprocessing wrapper

[skandermoalla]

Fair enough. I guess this makes this issue independent of Farama-Foundation/Arcade-Learning-Environment#467.