Mixed (Tate) motives in $A_g$