Cyclotomic factors of Serre's polynomials

These polynomials have deep connections with the theory of partition numbers and the Ramanujan $\tau$-function. They appeared for the first time in work of Newman 1955, and were used by Serre in his 1985 work on the lacunarity of the powers of the Dedeking eta function.  These polynomial have positive coefficients so all their real roots are negative. By the Euler pentagonal formula, it follows that $P_m(X)$ has a root at $-1$ for infinitely many $m$. We ask whether $P_m(X)$ can have other roots of unity except $-1$. We prove that this is never the case, namely that if $z$ is a root of unity of order $N\ge 3$ and $m\ge 1$, then $P_m(z)\ne 0$. The proof uses basic facts about finite fields and a bit of analytic number theory.