For Aristotle and Thomas Aquinas, for example, things in the world can change only if there is something that changes or actualizes everything else without the need (or indeed even the possibility) of its being actualized itself, precisely because it is already “pure actuality.” Change requires an unchangeable changer or unmovable mover.
For Neoplatonists, everything made up of parts can be explained only by reference to something that combines the parts. Accordingly, the ultimate explanation of things must be utterly simple and therefore without the need or even the possibility of being assembled into being by something else. Plotinus called this “the One.” For Leibniz, the existence of anything that is in any way contingent can be explained only by its origin in an absolutely necessary being.
But Krauss simply can’t see the “difference between arguing in favor of an eternally existing creator versus an eternally existing universe without one.” The difference, as the reader of Aristotle or Aquinas knows, is that the universe changes while the unmoved mover does not, or, as the Neoplatonist can tell you, that the universe is made up of parts while its source is absolutely one; or, as Leibniz could tell you, that the universe is contingent and God absolutely necessary. There is thus a principled reason for regarding God rather than the universe as the terminus of explanation.
One can sensibly argue that the existence of such a God has not been established. (I think it has been, but that’s a topic for another day.) One cannot sensibly dispute that the unchanging, simple, and necessary God of classical theism, if he exists, would differ from our changing, composite, contingent universe in requiring no cause of his own.