LLM-MRI Python Module: A brain scanner for LLMs
Resumo
arge Language Models (LLMs) have demonstrated remarkable capabilities, yet understanding how they encode and represent information internally remains a significant challenge. This work presents LLM-MRI, an open-source Python module designed to support LLM interpretability through the visualization and analysis of neural activation patterns. Inspired by magnetic resonance imaging, the tool maps high-dimensional activations into lower-dimensional representations, grouping neurons with similar activation patterns into regions. Its pipeline comprises three stages: activation extraction from a user-defined text corpus, neural region mapping through dimensionality reduction, and artifact generation. The module produces layer-wise activation maps, activation graphs connecting regions across adjacent layers based on co-activation, and graph visualizations that enable comparisons between document categories. These representations allow researchers to investigate whether different text categories activate distinct neural regions and to explore structural properties of activation patterns using complex network metrics. The module is designed to facilitate the investigation of how textual characteristics are reflected in the internal representations of Transformer-based LLMs. Currently under active development, LLM-MRI aims to expand its visualization capabilities and incorporate quantitative graph analysis to further support research into LLM interpretability.