A critical initialization for biological neural networks - Nature

Intrinsically generated neuronal activity shows coordinated macroscopic modes spanning the entire mouse brain. Variance is more concentrated in the top eigenmodes than expected from independent firing, without a low-dimensional cutoff. Variance concentration follows a power-law scaling with eigenmode number. Mechanistic explanations for this scaling are not yet available, but macroscopic variability has been linked to neural network dynamics in critical or chaotic regimes. Similar emergent macroscopic structure appears in artificial neural networks, where initialization can satisfy temporal requirements, accelerate learning, and improve final models. Common initialization methods scale weight amplitudes by inverse square roots of unit counts, while more complex schemes are less tested. Coordinated brainwide activity persists across timescales, including seconds-long patterns, potentially related to working memory, though the emergence of long timescales from fast neurons remains unclear.

"Intrinsically generated neuronal activity contains macroscopic modes of coordination between neurons that extend across the entire mouse brain2,3,4,5. More activity variance is concentrated into the top dimensions of neural activity than would be expected for independently firing neurons. At the same time, there is no low-dimensional cutoff of variance concentration, and variance scales as a power-law of the eigenmode number2. As yet, there are no mechanistic models that can explain this scaling of variance, but macroscopic variability in general has been hypothesized to arise from neural network dynamics operating in either a critical or chaotic regime6,7,8,9,10,11."

"Emergent macroscopic structure has also been studied in artificial neural networks, usually in the context of neural network initialization. Good initializations can directly satisfy the temporal requirements of many computational tasks12,13,14, or at least substantially accelerate subsequent learning and lead to better final models15,16,17. Commonly used initializations scale the amplitudes of the weight matrix by the inverse square root of the number of in-units, out-units or a combination of these15,18. More complex initialization schemes are rarely tested (but see refs. 19,20,21). A better understanding of emergent macroscopic structure may lead to better initialization schemes for modern, complex models such as transformers22, state space models23 and deep signal processing models24."

"Coordinated, brainwide neural activity has been observed across several timescales, including seconds-long patterns2,25. Such persistent activity has been hypothesized to form the basis for working memory26, but it is not well understood how the long timescales of working memory can emerge from individual neurons with fast dynamical properties. We follow previous work to assume that the interactions in a high-dimensional neural network can be approximated by random matrices27,28,29,30,31, which can summarize the"

Read at Nature