The modem is on a different subnet and not on the same lan, the brick is doing a kind of nat to present the modem with src IP addresses that it can cope with, since, lacking a gateway setting, the modem doesn't know how to communicate with anything that is outside its subnet range. Also it will not let me set the netmask to 0.0.0.0 not that that would necessarily help because the modem’s ARP lookups would just fail anyway presumably.
I'm wondering if SSH mentions its own source address in the protocol? Just guessing as I know nothing about SSH, have never looked it up.