In many real-world problems, one faces the problem of having to make decisions considering several conflicting objectives. In such problems, the solution is a set of policies rather than a single one. This leads to multi-objective reinforcement learning problems (MORL) which have not received much attention until recent years by the reinforcement learning community.

In this work, we propose coupling the R2 indicator with Pareto Q-learning. The R2 indicator has been successfully used for multi-objective optimization problems making it a good candidate for MORL. We tested our approach on several problems from MO Gymnasium and compared it with HB-MORL, which uses the hypervolume indicator. Our preliminary results show that the novel algorithm obtains competitive results and that could be an interesting alternative when dealing with MORL problems.