Arithmetic Structures in Spectral Models of Gravity