Limitations to equidistribution in arithmetic progressions