The Gottesman-Kitaev-Preskill (GKP) encoding of a qubit within an oscillator is particularly appealing for fault-tolerant quantum computing with bosons because Gaussian operations on encoded Pauli eigenstates enable Clifford quantum computing with error correction. We show that applying GKP error correction to Gaussian input states, such as vacuum, produces distillable magic states, achieving universality without additional non-Gaussian elements. Fault tolerance is possible with sufficient squeezing and low enough external noise. Thus, Gaussian operations are sufficient for fault-tolerant, universal quantum computing given a supply of GKP-encoded Pauli eigenstates.