Real-time Neural MPC: Deep Learning & Model Predictive Control Framework

Model predictive control (MPC) is a powerful technique that has gained popularity in recent years for controlling the movements of robotic agents. It involves using a model of the agent’s dynamics to predict its future behaviour. Followed by optimization of this behaviour to achieve a desired goal while also satisfying a set of constraints.

MPC algorithms are capable of handling complex scenarios and can consider various factors such as the agent’s physical constraints, environmental conditions, and the dynamics of other objects in the environment.

This makes MPC a popular choice for controlling the movements of robots in a variety of applications, including:

  • autonomous vehicles,
  • drones, and
  • industrial automation systems.

A real-time Neural MPC

In a combined effort, researchers at the Technical University of Munich and the University of Zurich have developed a real-time Neural MPC.

The framework combines advanced artificial neural networks (ANNs) with a model predictive control (MPC) framework to control the movements of agile robots, specifically quadrotors or drones with four rotors.

Data-Driven MPC for Quadrotors

The Robotics and Perception Group led by Davide Scaramuzza has been at the forefront of developing advanced control algorithms for robotic systems.

Their work – Data-Driven MPC for Quadrotors – has focused on integrating machine learning techniques, particularly deep learning, and reinforcement learning, with traditional control algorithms to improve the performance of robotic systems.

This became the reference study for Tim Salzmann and Markus Ryll, researchers at the Autonomous Aerial Systems Group of Technical University of Munich.

Scaramuzza’s approach as the starting point

The research team generalized Scaramuzza’s approach which uses Gaussian Processes (GPs), to generic Neural Networks (Deep Learning Models).

Eventually, they pitched the idea to the Robotics and Perception Group at the University of Zurich.

Researches envision the collaboration as the starting point to the development of new and innovative techniques for controlling robotic systems, with the potential to revolutionize many industries and applications.

Optimal actions for robots in real-time

The new framework proposed by the team combines deep learning models with online optimization of model predictive control (MPC) to enable real-time prediction of optimal actions for robots.

Deep learning models are known for their expressive power but can be computationally expensive to process.

However, the framework proposed by Salzmann, Ryll, and their colleagues approximates these models in real-time, allowing them to leverage dedicated hardware, such as GPUs, to efficiently process them.

This makes it possible to predict optimal actions for robots in real-time, enabling them to react quickly and adapt to changes in their environment.

Onboard optimal control of quadrotors

The use of PyTorch/TensorFlow on a GPU for deep learning computations, while performing control optimization in compiled C code on a CPU, have allowed for efficient and effective use of computational resources.

This is particularly useful in onboard optimal control of quadrotors, as it enables real-time prediction and control of the drone’s movements, explained Salzmann and Ryll.

The framework’s ability to combine deep learning with optimal control is particularly valuable because deep learning models can accurately predict the dynamics of a system. While optimal control algorithms can generate the optimal control signals required to achieve a desired goal.

By combining these two approaches, the Real-time Neural MPC framework has provide a powerful tool for controlling robotic systems in real-time.

Evaluation of real-time neural MPC framework

The Real-time Neural MPC framework was evaluated in both simulated and real-world environments and it gave optimistic results.

This is quite noteworthy. As it demonstrates the framework’s potential to control the actions of a highly agile quadrotor in real-time.

With a parametric capacity more than 4,000 times larger than those previously used to control the movements of agile robots in real-time is a significant achievement.

Furthermore, the finding that their framework can reduce positional tracking errors by up to 82% compared to conventional MPC methods without a deep learning component is equally fascinating.


Researchers have developed a robust framework. As mentioned before, it is doing remarkably well both in simulated and real-world settings.

The ability to predict optimal actions for robots in real-time can improve the efficiency, safety, and reliability of these systems, leading to significant benefits in a wide range of industries.

Once it’s out of lab, it has several potential applications, including in the field of autonomous vehicles, drones, and industrial automation systems.

Overall, the Real-time Neural MPC framework is a significant step forward in the field of robotics and control.

Via: TechXplore

Explore further