Replicator

Paul Adamski, M. Eleveld, Ankush Sood, Á. Kun, A. Szilágyi, T. Czárán, E. Szathmáry, S. Otto

Michael G. Howlett, A. Engwerda, R. Scanes, S. Fletcher

Daniel Hennes, Dustin Morrill, Shayegan Omidshafiei, Rémi Munos, J. Pérolat, Marc Lanctot, A. Gruslys, Jean-Baptiste Lespiau, Paavo Parmas, Edgar A. Duéñez-Guzmán, K. Tuyls

Policy gradient and actor-critic algorithms form the basis of many commonly used training techniques in deep reinforcement learning. Using these algorithms in multiagent environments poses problems such as nonstationarity and instability. In this paper, we first demonstrate that standard softmax-based policy gradient can be prone to poor performance in the presence of even the most benign nonstationarity. By contrast, it is known that the replicator dynamics, a well-studied model from evolutionary game theory, eliminates dominated strategies and exhibits convergence of the time-averaged trajectories to interior Nash equilibria in zero-sum games. Thus, using the replicator dynamics as a foundation, we derive an elegant one-line change to policy gradient methods that simply bypasses the gradient step through the softmax, yielding a new algorithm titled Neural Replicator Dynamics (NeuRD). NeuRD reduces to the exponential weights/Hedge algorithm in the single-state all-actions case. Additionally, NeuRD has formal equivalence to softmax counterfactual regret minimization, which guarantees convergence in the sequential tabular case. Importantly, our algorithm provides a straightforward way of extending the replicator dynamics to the function approximation setting. Empirical results show that NeuRD quickly adapts to nonstationarities, outperforming policy gradient significantly in both tabular and function approximation settings, when evaluated on the standard imperfect information benchmarks of Kuhn Poker, Leduc Poker, and Goofspiel.

Deependra Rastogi, Prashant Johri, M. Donelli, Seifedine Kadry, A. Khan, Giuseppe Espa, P. Feraco, Jungeun Kim

The most prevalent form of malignant tumors that originate in the brain are known as gliomas. In order to diagnose, treat, and identify risk factors, it is crucial to have precise and resilient segmentation of the tumors, along with an estimation of the patients’ overall survival rate. Therefore, we have introduced a deep learning approach that employs a combination of MRI scans to accurately segment brain tumors and predict survival in patients with gliomas. To ensure strong and reliable tumor segmentation, we employ 2D volumetric convolution neural network architectures that utilize a majority rule. This method helps to significantly decrease model bias and improve performance. Additionally, in order to predict survival rates, we extract radiomic features from the tumor regions that have been segmented, and then use a Deep Learning Inspired 3D replicator neural network to identify the most effective features. The model presented in this study was successful in segmenting brain tumors and predicting the outcome of enhancing tumor and real enhancing tumor. The model was evaluated using the BRATS2020 benchmarks dataset, and the obtained results are quite satisfactory and promising.

R. Mizuuchi, Taro Furubayashi, N. Ichihashi

In prebiotic evolution, self-replicating molecules are believed to have evolved into complex living systems by expanding their information and functions open-endedly. Theoretically, such evolutionary complexification could occur through successive appearance of novel replicators that interact with one another to form replication networks. Here we perform long-term evolution experiments of RNA that replicates using a self-encoded RNA replicase. The RNA diversifies into multiple coexisting host and parasite lineages, whose frequencies in the population initially fluctuate and gradually stabilize. The final population, comprising five RNA lineages, forms a replicator network with diverse interactions, including cooperation to help the replication of all other members. These results support the capability of molecular replicators to spontaneously develop complexity through Darwinian evolution, a critical step for the emergence of life. Long-term experimental evolution shows that a single polymerase-encoding RNA replicator can evolve into a complex replicator network, shedding light on how a molecular replicator could have developed complexity before the emergence of life.