Transition to chaos separates learning regimes and relates to measure of consciousness in recurrent neural networks

Summary

The critical coupling strength that separates chaotic from ordered dynamics in recurrent neural networks also differentiates two learning strategies: networks initialized with low coupling learn rich representations, while those with larger variance learn lazier solutions. Training moves both stable and chaotic networks closer to the edge of chaos. Biologically realistic connectivity fosters stability across a wide range of variances. The transition to chaos is reflected in the perturbational complexity index (PCIst), a measure that clinically discriminates levels of consciousness. Networks with high PCIst exhibit stable dynamics and rich learning, suggesting a consciousness prior may promote rich learning. The results indicate a relationship between critical dynamics, learning regimes, and complexity-based measures of consciousness.

Study at a glance

Abstract

Recurrent neural networks exhibit chaotic dynamics when the variance in their connection strengths exceed a critical value. Recent work indicates connection variance also modulates learning strategies; networks learn ”rich” representations when initialized with low coupling and ”lazier”solutions with larger variance. Using Watts-Strogatz networks of varying sparsity, structure, and hidden weight variance, we find that the critical coupling strength dividing chaotic from ordered dynamics also differentiates rich and lazy learning strategies. Training moves both stable and chaotic networks closer to the edge of chaos, with networks learning richer representations before the transition to chaos. In contrast, biologically realistic connectivity structures foster stability over a wide range of variances. The transition to chaos is also reflected in a measure that clinically discriminates levels of consciousness, the perturbational complexity index (PCIst). Networks with high values of PCIst exhibit stable dynamics and rich learning, suggesting a consciousness prior may promote rich learning. The results suggest a clear relationship between critical dynamics, learning regimes and complexity-based measures of consciousness.