$L_\infty$ algebras governing simultaneous deformations in algebra and geometry