Could have been a creep with a camera drone kitz, but from your description of its movement maybe more likely a drone someone had lost control over (out of range due to what ever reason) that went into GPS "loiter" mode before either regaining control or doing an auto return home.
I suggest this only as you mention it flitting slowly about, its what they do when in GPS position hold. Standard GPS will be accurate to a few meters, so any position hold will drift around in a couple of meter box.
Could also have been someone playing (recklessly) with autonomous way point navigation, i.e go to this far away place, hold, move to next etc.
Did you notice if the "front" part of it, or any particular "side" i.e with 2 rotors facing you next to each other, seemed to be deliberately facing you? That would be a give away that it was under someones control and they were looking at you, most cheap drones dont have cameras on gimbals to track without actually orienting the thing towards target.