Space Invaders Invisible bullets on Mode=0

[hyzhak]

I'm running this environment through the https://gymnasium.farama.org/environments/atari/space_invaders/ and found that even at the previous on the page above (Gymnasimum) there is no bullets from invaders (it is mode=0), on mode=1,4 bullets start be seeing. It doesn't like it supposed to be rendered.

space-invaders-invisible-bullets.mov

[pseudo-rnd-thoughts]

Could you provide a minimal example code to recreate this?

[hyzhak]

sure, here is the minimal example:

    import gymnasium as gym

    env = gym.make('ALE/SpaceInvaders-v5', render_mode='rgb_array')
    env = gym.wrappers.RecordVideo(env,
                                   video_folder='videos',
                                   episode_trigger=lambda episode: True,
                                   name_prefix="space-invaders",
                                   )
    env.reset(seed=42)
    terminated = False
    while not terminated:
        # do nothing
        states, rewards, terminated, truncated, infos = env.step(0)
        env.render()
    env.close()

btw, I found that it happens due to default value for frameskip=4 for this environment https://gymnasium.farama.org/environments/atari/space_invaders/, when I changed it to frameskip=1 (odd value) I have started seeing bullets -- because they are flickering between frames. As result default frameskip=4 makes agent blind to bullets. There could be different solution thought, e.g. avg/max pulling of frames during frameskiping, as it was requested here: #467. Or changing default frameskip from 4 to some odd value.

[pseudo-rnd-thoughts]

Yes, I suspected that this would be the issue
This is why the wrappers.AtariPreprocessing requires a frameskip of 1
But if you apply the record video after the preprocessing wrappers you will have a similar issue that the rendered frames will most likely not include the bullets

[hyzhak]

find one more ALE environment with similar issue https://gymnasium.farama.org/environments/atari/frogger/
it could be seen right on the gif at the top of the page -- when frog leaps at the line with vehicles on it they disappear, but if you can check videos on youtube of this game -- they are just starting flipping on odd frame

[pseudo-rnd-thoughts]

Thanks, I believe these are known Atari 2700 Rom optimisations, certainly for Space Invaders, not certain for frogger but wouldn't be surprised.
The problem is that due to these being intentional features (optimisations) within the Roms, these aren't bugs in the usual form so I don't think we can fix them

[hyzhak]

Yes, I aware about this optimisation and I'm not calling to fix Atari simulation but rather environment itself, like the proposal in the issue #467 by averaging few frames.

1st as a humans we don't see individual frames but blend a few in a raw and so it shouldn't hurt agent either
2nd the current ALE environments exclude few frames which make invisible part of the scene in many environments (which human player can see but agent cannot). For some environments it makes training very hard if possible since agent doesn't receive necessary state.

Quick and dirty solution would be to set frameskip to 0, average frames on client side and ignore ALE environment optimisation but it will likely degrade significantly performance of emulation (as it was mentioned in #467).

There are few other examples (and I guess I can find more flaws like this in other environments, since flipping frames was the common optimization in Atari games):

Asteroid

gymnasium

Framekip=0

asteroids-fs1-episode-0.mp4

agent can see asteroids

import gymnasium as gym

env = gym.make("ALE/Asteroids-v5",
               render_mode="rgb_array",
               frameskip=1,
               )

env = gym.wrappers.RecordVideo(env,
                               video_folder='videos',
                               episode_trigger=lambda episode: True,
                               name_prefix="asteroids-fs1",
                               )
env.reset(seed=42)
terminated = False
while not terminated:
    # do nothing
    states, rewards, terminated, truncated, infos = env.step(0)
    env.render()
env.close()

Framekip=4

asteroids-fs4-episode-0.mp4

agent see no asteroids (which can be seen on gymnasium page as well)

import gymnasium as gym

env = gym.make("ALE/Asteroids-v5",
               render_mode="rgb_array",
               frameskip=4,
               )

env = gym.wrappers.RecordVideo(env,
                               video_folder='videos',
                               episode_trigger=lambda episode: True,
                               name_prefix="asteroids-fs4",
                               )
env.reset(seed=42)
terminated = False
while not terminated:
    # do nothing
    states, rewards, terminated, truncated, infos = env.step(0)
    env.render()
env.close()

Pacman

gymnasium

Framekip=0

pacman-fs1-episode-0.mp4

agent see all ghosts

import gymnasium as gym

env = gym.make("ALE/Pacman-v5",
               render_mode="rgb_array",
               frameskip=1,
               )

env = gym.wrappers.RecordVideo(env,
                               video_folder='videos',
                               episode_trigger=lambda episode: True,
                               name_prefix="pacman-fs1",
                               )
env.reset(seed=42)
terminated = False
while not terminated:
    # do nothing
    states, rewards, terminated, truncated, infos = env.step(0)
    env.render()
env.close()

Framekip=4

pacman-fs4-episode-0.mp4

agent see only one ghost (the rest are invisible, which can be seen on gymnasium page as well)

import gymnasium as gym

env = gym.make("ALE/Pacman-v5",
               render_mode="rgb_array",
               frameskip=4,
               )

env = gym.wrappers.RecordVideo(env,
                               video_folder='videos',
                               episode_trigger=lambda episode: True,
                               name_prefix="pacman-fs4",
                               )
env.reset(seed=42)
terminated = False
while not terminated:
    # do nothing
    states, rewards, terminated, truncated, infos = env.step(0)
    env.render()
env.close()