Spectrally rigid subsets of free groups