Ferrari vs Red Bull, race strategy: the value of Reinforcement Learning

Race strategy in Formula 1 can determine the outcome of a race when one’s car is competitive enough. This has been a topic of discussion since the inception of the premier motorsport category, and over time, teams have expanded their departments dedicated to strategic management. Back in 2009, for example, Red Bull underwent a significant “restyling” of this department, appointing individuals with extensive experience in game theory to lead the personnel.

This move was made because, in essence, the concept is quite similar. Thus, teams began using the so-called “Monte Carlo Simulations.” This is a probabilistic method that simulates multiple races lap by lap, varying the scenario each time. In other words, for each lap, different scenarios are analyzed based on various factors, such as a Safety Car, Virtual Safety Car, or yellow flag. The aim is to predict what might happen.

These simulations are still in use and are generally combined with other types of predictions. Hypothetical scenarios are often initiated during the race, entering various inputs into the computers based on what is happening on the track. This way, with reduced accuracy, one can attempt to predict what might happen to their driver in the following laps.

Since the 2017 season, F1 teams have considered the use of artificial intelligence, a science that analyzes and identifies patterns through managing large amounts of data. The software that simulates various race situations through “machine learning” is known as “Race Planner.” Currently, a further step has been taken, incorporating “Reinforcement Learning,” a method where labeling each data point and defining what is “right” or “wrong” is unnecessary.

This is a highly advanced technique that adopts a significantly different method in “training the model.” The significant advantage of this practice lies in its self-learning capability, useful for acquiring complex behaviors and offering recommendations for both the short and long term. In other words, Reinforcement Learning is a technique through which a computer (agent) learns to perform an activity or task through repeated “trial-and-error” interactions.

This approach allows the agent to make decisions that can maximize the on-track position without human intervention. For example, the agent may choose a short-term path during the race that involves losing some positions but regaining them later in the race. Moreover, as mentioned, by learning through trial and error, no input of information labeling the data is required, as this will happen automatically.

F1/Ferrari vs Red Bull, Race Strategies: Reinforcement Learning
Continuing, let’s introduce the concept of “control.” Through Reinforcement Learning, two tasks can be accomplished based on the applied “policy.” This term refers to a function and a set of rules specifying which action the agent should take in a given situation to maximize a predefined objective known as “reward.” In other words, it’s a decision-making strategy that the agent follows to select actions.

Show your support for Scuderia Ferrari with official merchandise collection! Click here to enter the F1 online Store and shop securely! And also get your F1 tickets for every race with VIP hospitality and unparalleled insider access. Click here for the best offers to support Charles and Carlos from the track!

The first is called “prediction,” where the policy is defined and fixed, aiming to predict the effectiveness of a certain strategy. This is essentially the old approach that F1 had in the past. Until 2015, teams selected some strategies, trying to understand which one was the fastest or, generally, the best. They measured the mere adaptation of a given policy to optimize the reward provided by the sequence of actions taken.

Everything becomes more complex and interesting when we apply the concept of “control.” In this case, the agent’s goal is to find the optimum, namely the self-definition of a policy that allows, considering any situation on the track, the achievement of the most suitable scenario possible.

Applying this concept to F1, control analyzes and finds different complicated patterns through data collection, adapting in real-time to what is happening. Let’s take an example and assume we are in the following race state (referred to as ‘state’). To do this, we’ll use an image from the following, trying to understand how these measures work.

A “state” is defined by parameters such as the current lap number, the instantaneous ranking, the gaps between drivers, and the tires each driver is using with the respective number of laps for each tire set. At this point, a race can be formulated through the Markov Decision Process (MDP), where the probability of transitioning from one state to another is based solely on the last observed state.

In our example depicted in the previous shot, we are talking about the state of lap 11. Therefore, laps 1 to 10 have a lesser weight on what will happen in the next state from lap 12 onwards. Of course, as accurate as the model may be, it will never definitively predict what will happen in the next state, mainly because many on-track events are “stochastic.”

To simulate the next state, the potential pace of each driver must first be estimated. This can be done through a function that takes into account the tire compound, the tire’s behavior, the tire model, and the decreasing fuel mass over laps. Each lap introduces an update to observable variables, such as the actual car pace, tire age, the gap between drivers, and everything else.

In predicting the next passage, the agent must also consider the “overtake model,” a model that takes into account the probability of each driver being overtaken or overtaking. The basic model takes various inputs for each driver, such as the gap from those ahead and those behind, the potential pace of other competitors, and a coefficient indicating the ease of overtaking an opponent. Additionally, the “pit lane time” must be evaluated, which is the time taken to traverse the entire pit lane during a pit stop.

Now, suppose the agent is analyzing the current race state, trying to predict the “next lap” state by choosing the most effective policy to use. Recall that we are considering the task of Control, not Prediction. Q-Learning is a technique used in RL to choose the policy to follow.

For each state, it is possible to assess the impact that a certain action, determined by a specific policy, would have. Each state-action pairing can be associated with a certain “Q-Value.” This way, the agent can decide which policy to use by selecting the state-action pairing with the highest Q-Value.

F1, Race Strategies: Red Bull vs Ferrari
The strength of RL lies precisely in not relying solely on historical data but interacting with the working environment. In general, this is not always possible, but in F1, it becomes feasible. Currently, Red Bull has an advantage in this field, even though other teams implement “prediction” using these techniques. The first partnership with Oracle, specialized in data analysis through artificial intelligence, dates back to 2021. However, it’s from the following year that the U.S. company becomes the main sponsor of the Milton Keynes team.

This is a close collaboration that has allowed continuous improvement in the manipulation and use of this data, giving the Austrian team a significant advantage over its competitors. Essentially, during the race, Red Bull can predict what will happen in the next 2-3 laps with high accuracy, which is undoubtedly challenging to achieve.

Furthermore, in Milton Keynes, they are already using neural networks. These are a subset of machine learning and represent the central element of deep learning algorithms. Their name and structure are inspired by the human brain, mimicking how biological neurons send signals. Analyzing the chart above, taken directly from the database of the American company Oracle, showing the results of their investigation into different types of approaches.

In each of these scenarios, they were asked to determine the winner of more races, and the highest score was obtained by the neural network, which seems to be the most accurate at the moment. To be clear, other teams have not been idle. Already in 2020, James Vowles, then director of Mercedes’ strategies, stated that the German team had been using artificial intelligence for most aspects related to race strategy for some time.

Indeed, in 2018, during the era of three-pointed star dominance, the Brackley team began delving into “Big Data.” Only in 2022 did the team understand the importance of having a strategic partner, establishing a partnership with G42, a company specializing in artificial intelligence. In all of this, where does Ferrari stand? Well, the Prancing Horse has also participated in this trend and in 2021 signed an agreement with AWS, a branch of the Amazon giant dedicated to Machine Learning.

In practice, however, the Maranello team announced AWS as a “cloud Partner.” The Modena-based team currently uses Amazon Simple Storage Service (Amazon S3) and AWS Lake Formation to quickly and securely collect, catalog, and clean hundreds of petabytes of data.

At the time, Mattia Binotto stated that this collaboration would give rise to a “data-driven” organization when other teams had possessed it for years. However, the various AWS models should serve not only to analyze and make decisions related to race strategy but also to better understand the data coming from the track itself, such as temperature, lap times, and grip levels. Obviously, the difference lies in the accuracy of manipulation, and, apparently, considering the often unsuccessful strategies of the Scuderia, there don’t seem to be models capable of predicting the outcome of the following laps during the race.