Discovering state-of-the-art reinforcement learning algorithms

"By contrast, artificial agents typically learn using hand-crafted learning rules. Despite decades of interest, the goal of autonomously discovering powerful RL algorithms has proven elusive7-12. In this work, we show that it is possible for machines to discover a state-of-the-art RL rule that outperforms manually-designed rules. This was achieved by meta-learning from the cumulative experiences of a population of agents across a large number of complex environments. Specifically, our method discovers the RL rule by which the agent's policy and predictions are updated."

"In our large-scale experiments, the discovered rule surpassed all existing rules on the well-established Atari benchmark and outperformed a number of state-of-the-art RL algorithms on challenging benchmarks that it had not seen during discovery. Our findings suggest that the RL algorithms required for advanced artificial intelligence may soon be automatically discovered from the experiences of agents, rather than manually designed."

Powerful RL mechanisms in biological systems were produced by evolution through trial and error. Machines can discover such mechanisms via meta-learning from the cumulative experiences of many agents across diverse complex environments. The discovery focuses on an update rule that governs policy and prediction updates. Large-scale experiments show the discovered rule outperforms existing hand-designed rules on the Atari benchmark and generalizes to outperform state-of-the-art RL algorithms on unseen challenging benchmarks. The results indicate that advanced RL algorithms can emerge automatically from population-level experience rather than rely solely on manual algorithm design.

Read at www.nature.com