From 7087d48280ff715dac46a531702903b9aa71f986 Mon Sep 17 00:00:00 2001 From: Pranav Gupta Date: Fri, 23 Aug 2024 09:51:29 +0100 Subject: [PATCH] Documentation of new Mountain Car environments (#101) Co-authored-by: Lucas Alegre --- docs/environments/classical.md | 10 +++++++++- mo_gymnasium/envs/mountain_car/mountain_car.py | 2 +- 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/docs/environments/classical.md b/docs/environments/classical.md index 3f80ce85..0d15285e 100644 --- a/docs/environments/classical.md +++ b/docs/environments/classical.md @@ -9,9 +9,17 @@ Multi-objective versions of classical Gymnasium's environments. | Env | Obs/Action spaces | Objectives | Description | |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------|---------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [`mo-mountaincar-v0`](https://mo-gymnasium.farama.org/environments/mo-mountaincar/)
| Continuous / Discrete | `[time_penalty, reverse_penalty, forward_penalty]` | Classic Mountain Car env, but with extra penalties for the forward and reverse actions. From [Vamplew et al. 2011](https://www.researchgate.net/publication/220343783_Empirical_evaluation_methods_for_multiobjective_reinforcement_learning_algorithms). | -| [`mo-mountaincarcontinuous-v0`](https://mo-gymnasium.farama.org/environments/mo-mountaincarcontinuous/)
| Continuous / Continuous | `[time_penalty, fuel_consumption_penalty]` | Continuous Mountain Car env, but with penalties for fuel consumption. | +[`mo-mountaincar-3d-v0`](https://mo-gymnasium.farama.org/environments/mo-mountaincar/) **
| Continuous / Discrete| `[time_penalty, move_penalty, speed_objective]` | The forward and backward penalties have been merged into the move penalty and a speed objective has been introduced which gives the positive reward equivalent to the car's speed at that time step.* | +[`mo-mountaincar-timemove-v0`](https://mo-gymnasium.farama.org/environments/mo-mountaincar/) **
| Continuous / Discrete | `[time_penalty, move_penalty]`| Class Mountain Car env but an extra penalty for moving backwards or forwards merged into a move penalty. | +[`mo-mountaincar-timespeed-v0`](https://mo-gymnasium.farama.org/environments/mo-mountaincar/) **
| Continuous / Discrete| `[time_penalty, speed_objective]` | Class Mountain Car env but an extra positive objective of speed which gives the positive reward equivalent to the car's speed at that time step.* +| [`mo-mountaincarcontinuous-v0`](https://mo-gymnasium.farama.org/environments/mo-mountaincarcontinuous/)
| Continuous / Continuous | `[time_penalty, fuel_consumption_penalty]` | Continuous Mountain Car env, but with penalties for fuel consumption. | | [`mo-lunar-lander-v2`](https://mo-gymnasium.farama.org/environments/mo-lunar-lander/)
| Continuous / Discrete or Continuous | `[landed, shaped_reward, main_engine_fuel, side_engine_fuel]` | MO version of the `LunarLander-v2` [environment](https://gymnasium.farama.org/environments/box2d/lunar_lander/). Objectives defined similarly as in [Hung et al. 2022](https://openreview.net/forum?id=AwWaBXLIJE). | +*An additional objective was introduced to prevent the agent from converging to the local maxima due to a lack of reward signal for the static action. + +**Read more about these environments and the detailed reasoning behind them in [`Pranav Gupta's Dissertation`](https://drive.google.com/file/d/1yT6hlavYZGmoB2phaIBX_5hbibA3Illa/view?usp=sharing) + + ```{toctree} :hidden: :glob: diff --git a/mo_gymnasium/envs/mountain_car/mountain_car.py b/mo_gymnasium/envs/mountain_car/mountain_car.py index 0c2c5dbb..f49cf5ce 100644 --- a/mo_gymnasium/envs/mountain_car/mountain_car.py +++ b/mo_gymnasium/envs/mountain_car/mountain_car.py @@ -19,7 +19,7 @@ class MOMountainCar(MountainCarEnv, EzPickle): - reverse penalty: -1.0 for each time step the action is 0 (reverse) - forward penalty: -1.0 for each time step the action is 2 (forward) - Alternatively, the reward can be changed with the following options: + #Alternatively, the reward can be changed with the following options: - add_speed_objective: Add an extra objective corresponding to the speed of the car. - remove_move_penalty: Remove the reverse and forward objectives. - merge_move_penalty: Merge reverse and forward penalties into a single penalty.