Conclusions about Neural Network to Brain Alignment are Profoundly Impacted by the Similarity Measure
Deep neural networks are popular models of brain activity, and many studies ask which neural networks provide the best fit. To make such comparisons, the papers use similarity measures such as Linear Predictivity or Representational Similarity Analysis (RSA). It is often assumed that these measures yield comparable results, making their choice inconsequential, but is it? Here we ask if and how the choice of measure affects conclusions. We find that the choice of measure influences layer-area correspondence as well as the ranking of models. We explore how these choices impact prior conclusions about which neural networks are most "brain-like". Our results suggest that widely held conclusions regarding the relative alignment of different neural network models with brain activity have fragile foundations.