PyCharm

OpenAi Gym comes with many examples environments for learning how to use reinforcement learning algorithms, including MountainCar, Acrobot and some Atari games such as Space Invaders. However for our Hello World project, we’ll have a look at the classic Cart Pole problem. The aim of Cart Pole is to try and balance the pole above the cart for as long as possible by moving the cart left and right.

So let’s walk through the implementation step by step and I’ll explain what we are doing as we go along. First of all in PyCharm create a new python script (right click in project (left panel) -> New -> python file) or delete the contents of the default Python script created by PyCharm. Now we want to include the packages required for OpenAi Gym and the ML algorithms. import gym from stable_baselines3 import DQN Adding the above code to the top of the python script will import the OpenAi Gym module (import gym) and Deep Q Network (DQN) ML algorythm (from stable_baseline3 import DQN) into the project. Now we can go ahead and create our Gym environment by adding the following line environment = gym.make("CartPole-v0") Now we are able to run the script for the first time. In PyCharm you can press the play button in the top right of the main window (assuming you used the default script that PyCharm creates, otherwise click on the dropdown labelled “main“ -> “Edit configuration“ and change the “script path“ to match the script you are working on). You should notice that nothing happens, however you should also notice that the application does not exit, so this means something is happening. The reason we don’t see anything is because we have not told the environment to render the output to screen yet, but before we can do that we could do with training our agent, so let’s have a look at how we can do that next.   model = DQN("MlpPolicy", environment, verbose=1)
model.learn( total_timesteps=100_000 )
model.save("models/dqn_cartpole")
So what does this do,

• model = DQN("MlpPolicy", environment, verbose=1)
This creates a new Deep Q Network ML model, in which we can be trained to play the CartPole environment. As we can see, it contains three parameters of which the first two are required. MlpPolicy is the learning policy, environment is the OpenAI Gym environment that we want to train our agent in. The third parameter verbose=1 is optional and when it is set to 1 (ie =1) it prints the learning statistics to the development console, while setting it to 0 will print nothing. We’ll set it to 1 so we know something is happening.
• model.learn( total_timesteps=100_000 )
This tells our ML model to start learning our environment. It achieves this by exploring the environment through a series of trial and error actions based on the current observations of the environment. You’ll notice that we have included one optional parameter total_timesteps this is the number of timesteps (or updates/ticks) that the ML model will train for. We’ll set this to 100,000 for now (the _ is just a way to space out numbers in python which I think is a nice feature of the langerage) and you can play around with this parameter in your own time. What happens if you increase or decrease total_timesteps? Lastly we have,
• model.save("models/dqn_cartpole")
This simply saves the ML model so we can load it back in at a later time. It contains a single parameter for the file that data should be saved in, relative to the running script. ie. C:/user/username/python Projects/My First ML/models/dgn_cartpole

Now if we run the script again we should notice that it starts printing an output to the development console of PyCharm (at the bottom). The output is just the statistics of the ML Model, it just lets us know that it is trying to learn something. if we leave this running for a few minutes you’ll notice that the application will just exit (and create a new save file). If you don’t want to wait you can either press the stop button or click in the development console and press ctrl+c. Again though we still don’t see our environment (or game) yet, this is because rendering is usually the slow part of running a game, and we want to train our agents as fast as possible. The only time you will need to render the game during training is if you wanted to use machine vision to play the game, but that’s another post for another day.

Once the agent has finished training, we are able to let it play the game by itself (at least to some degree). This is where we can actually render the environment and see how well the agent has done. First of all we need to define a couple of variables episodes = 100 update_steps = 1000 The first one episodes is the amount of times the agent can reset the environment, upon death or completion of the scene. In this case we have set it to 100. The second one update_steps is the maximum number of updates (or ticks) before the scene is automatically reset. Now it’s time to render the environment. This is probably the trickiest part, but don’t worry we go through it line by line.
for episode in range(episodes):

  for step in range(update_steps):
    action, _state = model.predict( observation, deterministic=True )
    observation, _reward, done, info = environment.step( action )
  environment.render()

  if done:
    print( f"Ended environment after {step} steps" )
    print(f"last info: {info}")
    observation = environment.reset()
    break

Starting from the top

• for episode in range(episodes):
is a loop and we are basically saying run the following indented code “episodes“ amount of time. (Note that the indentation is very important in Python)
• for step in range(update_steps):
we are doing the same as above but for update_steps amount of time. This basically means that we are going to tick the environment upto episodes * update_steps (or 100,000 with our configuration) amount of times (assuming it does not reset early).
• action, _state = model.predict( observation, deterministic=True )
Here we are asking the ML model to predict the next action and state of the environment. However we only need the action, so we put an underscore (_) in front of the state to show that we are discarding this value. Alternatively we could do action = model.predict( observation, deterministic=True )[0] witch means we only want the first value.
• observation, _reward, done, info = environment.step( action )
Next we update/tick the environment, witch returns four values.
• observation is the state of the environment at the end of the tick.
• _rewardis the reward that the agent would have received if we were training.
• done has agent finished (ie dead or completed the scene)
• infoany debug info generated during the tick.
• environment.render()
this renders the frame to the output window (see i told yay we’ll get there soon :D)
• if done:
here are asking if “done“ == “True“. If it is run the following indented code.
• the next two lines just prints a message to the output console the first print the amounts of ticks that the agent survived while the second print the last frames debug info.
• observation = environment.reset()
simply resets the scene back to its initial state, ready for the agent to make another attempt.

And that’s pretty much it! Now if you hit the play button in PyCharm, it will start the learning process, once it completes the environment windows should appear with the agent attempting to balance the pole for as long as it can!
The only thing i haven’t told you yet is how to load in your saved ML model.
To do that replace
model = DQN("MlpPolicy", environment, verbose=1 )
model.learn( total_timesteps=100_000 )
model.save("models/dqn_cartpole")

with
model = DQN.load(f"models/dqn_cartpole")