The only moving parts on a GPU are the fans that are included on it. It is otherwise a purely electronic design with no moving parts, so I don't really understand what you mean by this.
In terms of SSDs, they have no moving parts either, yes, but as I said, a graphics card is also a plug and play device. Most CPUs have one included nowadays anyway, albeit not very capable. The reason we use dedicated GPUs is because the CPU integrated ones are bound by thermal and power limits of the motherboard, and the system RAM being slow, whereas a dedicated GPU has access to more power and cooling, as well as having much higher bandwidth memory (3-4x bandwidth).