As has already been discussed certain chips do work best on certain lines... so I'll try not to bore you any further on that subject
and I'll attempt to answer your original questions.
As long as you have the latest firmware on the HG612, then there isnt any other way to push the HG612 to match the HH5. Therefore I can fully understand why you wish to retain the HH5A as your modem and therefore this would be your best scenario
HH5 -> 7800N However there is a known issue with the HH5A where it does on occasion drop both wifi and dsl connection. There was supposed to be a
fix rolled out, but you still see the odd mention of it.
What concerns me is Im unsure if when you attach the 7800N to the HH5A if you will still see the disconnection issue. It will entirely depend on whether its modem or wifi related.
However since you already have this equipment, then there is certainly no harm in trying it and it may do the job.
Unfortunately I dont have a HH5A and no idea how to put it into modem only mode. Perhaps one of the other guys can help you with this and tell you what you need to do.