Initial approach to create a controller for TORCS by learning how another controller
Initial approach to create a controller for TORCS by learning how another controller
or humans play the game.
The data obtained from 3 controllers.
One human player and two controllers:
The winner of the WCCI 2008 Simulated Car Racing
Hand coded controller that performs a complete lap in all tracks.
First, each kind of controller is imitated separately, then a mix of data is used to
First, each kind of controller is imitated separately, then a mix of data is used to
create new controllers.
The imitation is performed by means of training a feed forward neural network with the data, using the backpropagation algorithm for learning.
Human players realize they are not playing vs. another human – and finds a way to beat the NPC (non-player character).
Human players realize they are not playing vs. another human – and finds a way to beat the NPC (non-player character).
NPC sometimes cheats for winning humans.
Another option is to play in Internet against other human players
With a lot of cheats or playing versus experienced human make you lose in every game – boring!
Create opponents as intelligent as a human player.
Create opponents as intelligent as a human player.
The AI must be able to adapt its behavior depending on the opponent (play in the same level of the human)
In this way the AI will provide a better entertainment for the player!
Create competitive NPCs that imitates the human behavior.
Create competitive NPCs that imitates the human behavior.
The controller can play as well as its opponent – in the same level.
NPC can adapt its adapt its behavior when the human improve his/her player skills to remain competitive.
Realistic game where human plays versus one or more NPC.
There is option to compare the results with other researchers.
Allows to analyze behaviors that take place in a short period of time.
In all the experiments the controller created is a feed-forward ANN (Artificial Neural Networks) that was trained with data generated by the controllers.
In all the experiments the controller created is a feed-forward ANN (Artificial Neural Networks) that was trained with data generated by the controllers.
The learning algorithm for the ANN was backpropagation.
Wide area of researching is to create computational intelligence in games with ANN.
Wide area of researching is to create computational intelligence in games with ANN.
NEAT – NeuroEvolution of Augmenting Topologies
NEAT is an effective method (algorithm) to create ANN - it alters both the weighting parameters and structures of networks.
It starts with a small population of random ANN (with only input & output layers) that evolves to the problem.
An approaches to adapt the AI of the game to the player
An approaches to adapt the AI of the game to the player
Examples:
Rapidly adaptive game AI – method that applies continuously small adaptations to the AI based on observations and evaluation of the user actions.
Dynamic Scripting – based on a set of rules that are used for the game, whose weights to select one or another rule are modified through a machine learning algorithm.
One researcher clone the behavior of RoboCup player using case base reasoning (solving new problems based on the solutions of similar past problems)
One researcher clone the behavior of RoboCup player using case base reasoning (solving new problems based on the solutions of similar past problems)
Other researcher program robosoccer agents by modelling human behaviors with successful results.
Other researcher program robosoccer agents by modelling human behaviors with successful results.
1. Introduction
1. Introduction
2. Related work
3. TORCS competition
4. Controllers
5. Controller learning by imitation
6. Results
7. Conclusions
8. Future works
Very realistic simulator that has a sophisticated physic engine that takes into account many aspects of the racing such as fuel consumption, collisions or traction.
Very realistic simulator that has a sophisticated physic engine that takes into account many aspects of the racing such as fuel consumption, collisions or traction.
TORCS is open software and that allows the researchers to make modifications to the game and adapt it to their requirements.
Info. Provided:
Info. Provided:
The lap (current lap time, best lap time, distance raced, race position)
The car status (damage, fuel, actual gear, speed, lateral speed and R.P.M)
Distanse between the car and the track edges
More..
1. Introduction
1. Introduction
2. Related work
3. TORCS competition
4. Controllers
5. Controller learning by imitation
6. Results
7. Conclusions
8. Future works
In the experiments we use the data obtained from three different controllers:
In the experiments we use the data obtained from three different controllers:
Human player
The winner of the WCCI2008 competition
Hand coded controller
The information that the human gets from watching the game monitor is much richer.
The information that the human gets from watching the game monitor is much richer.
He drove the car trying to go through the middle of the road, with soft accelerations and brakes, braking much before the next curve started, without fast and sharp turns.
The human tried to drive the car as programmed controller would do, but with the mistakes that human makes.
Created my Matt Simmerson. This controller was the winner of the WCCI 2008 simulated car racing competition.
Created my Matt Simmerson. This controller was the winner of the WCCI 2008 simulated car racing competition.
As INPUTs of the ANN created by NEAT he selected: the current speed, the angle to track axis, the track position with respect to left and right edges, the current gear selection, the four wheels spin sensors, the current R.P.M and 19 track sensors.
All these inputs were scaled to the range [0,1].
The OUTPUTs of the ANN were:
The OUTPUTs of the ANN were:
The power (accelerate and brake), the gear change and the steering.
The two first are range [0,1] and the last one is in range [-1,1].
The fitness function used to evaluate the ANN in NEAT took into account the distance raced, the R.P.M, the maximum speed reached and a value to measure of how much the car stayed on the track.
The idea of creating another controller was due to the human controller sometimes make mistakes and the Simmerson’s controller does not perform one complete lap in all the tracks and sometimes gets out from the track.
The idea of creating another controller was due to the human controller sometimes make mistakes and the Simmerson’s controller does not perform one complete lap in all the tracks and sometimes gets out from the track.
Thus, the requirements are:
To have same outputs for same inputs (without mistakes) – deterministic
To calculate the values for the acceleration and the brake, we calculate the speed the car should have (estimated speed).
To calculate the values for the acceleration and the brake, we calculate the speed the car should have (estimated speed).
Where sum_semsors is the sum of the three front sensors (which give the distance between the car and the edge of the track).
Alpha and beta are predefined parameters.
With this value (estimated speed) we calculated the diffrence:
The acceleration and brake values are proportional to the absolute value of the difference of the actual speed and the estimated speed:
The acceleration and brake values are proportional to the absolute value of the difference of the actual speed and the estimated speed:
Where again there are adjustment parameters.
The steering value calculation:
The steering value calculation:
First, we check if the car is in straight or in a curve. We suppose that the car is in a straight when any of the 3 front sensors has the max value, and in a curve otherwise.
Straight equation:
Curve equation:
Finally the gear is calculated by:
Finally the gear is calculated by:
Where lambda is an adjustment parameter and speed is the current speed of the car.
(for the gear the controller does not allow to change the gear twice in less than a second)
1. Introduction
1. Introduction
2. Related work
3. TORCS competition
4. Controllers
5. Controller learning by imitation
6. Results
7. Conclusions
8. Future works
For the goal of learning the behavior of the controller we have used an ANN.
For the goal of learning the behavior of the controller we have used an ANN.
First, we obtain the data from the controller we want to imitate, then the ANN is trained with the backpropagation algorithm and finally the new controller is tested in the tracks.
Inputs:
Inputs:
Current speed of the car
Angle of the car with the axis
The current gear
The lateral speed of the car
The R.P.M
The 4 spins of the wheels.
19 sensors – distance between the car and the adges of the track
For all the experiments the ANN has 3 hidden layers of 28 neurons each one, were trained during 1000 cycles and the learning rate starts in 0.9 and finished in 0.0001.
For all the experiments the ANN has 3 hidden layers of 28 neurons each one, were trained during 1000 cycles and the learning rate starts in 0.9 and finished in 0.0001.
The data was taken from 17 roads tracks (next slide) – but only if the controller complete almost 3 laps.
That have been obtained per each controller:
That have been obtained per each controller:
1. Introduction
1. Introduction
2. Related work
3. TORCS competition
4. Controllers
5. Controller learning by imitation
6. Results
7. Conclusions
8. Future works
For the “controllers learning by imitation” we used the data of all tracks to train the ANN of the controller and then test it in each track.
For the “controllers learning by imitation” we used the data of all tracks to train the ANN of the controller and then test it in each track.
The time obtained by the controllers described before each track
The time obtained by the controllers described before each track
The results of the learnt controllers for all the 3-controllers.
The results of the learnt controllers for all the 3-controllers.
A controller created with the data of human, Simmersons and handcoded controllers was not created because they did not get good result with mixed configurations, as shown in the last 3 tables.
A controller created with the data of human, Simmersons and handcoded controllers was not created because they did not get good result with mixed configurations, as shown in the last 3 tables.
1. Introduction
1. Introduction
2. Related work
3. TORCS competition
4. Controllers
5. Controller learning by imitation
6. Results
7. Conclusions
8. Future works
It is very complicated to learn the human behavior in a video game:
It is very complicated to learn the human behavior in a video game:
The human makes different actions in same situations.
He does not make all actions in the proper way.
He makes a lot of mistakes that have to solve.
This sort of data makes completely impossible that an ANN could learn something useful.
The gear problem: the gear change has not been learned, despite it is probably the easiest output to learn. Maybe it is because of the high amount of data and due to the gear is also an input of the ANN.
The gear problem: the gear change has not been learned, despite it is probably the easiest output to learn. Maybe it is because of the high amount of data and due to the gear is also an input of the ANN.
The mixed data from two types of controllers – the result show that those controllers do not work.
The controller learned has mixed features, but non of these is learned properly, so the car goes out of the track easily and the simulation ends.
1. Introduction
1. Introduction
2. Related work
3. TORCS competition
4. Controllers
5. Controller learning by imitation
6. Results
7. Conclusions
8. Future works
Pre-process of the data before use it to train the neural network. Two ideas:
Pre-process of the data before use it to train the neural network. Two ideas:
To decrease the amount of data / remove duplicates.
Data pattern with same input and different output must be removed.
Train the controllers with some data of one controller.
Train the controllers with some data of one controller.
For example, the data of the straights of one controller that performs the straights good and the data of the curves of another controller that make well the turns.
If we want to imitate the human behavior:
If we want to imitate the human behavior:
We have to increase the information used to train the ANN. There is a lack of information because the human player senses more information from the domain that the other controllers use. The human also remember and improve his/her behavior in each lap.
The human also not perform in the same action under the same circumstances.
The ANN:
The ANN:
There is no context information, the controller does not remember its past actions and cannot take decision with that information.
Maybe recurrent neural networks need to be used and maybe we have to try different learning algorithms.
