Yes. Provide the option to replace them with modem-only devices under the operator's control if there are concerns.
It's standard for point to point fibre providers to provide the media converter to the customer, though I would go beyond that. Where either PON or HFC are used the operator controlling the equipment and presenting Ethernet to the customer, at the customer's discretion, is a good idea.
TL;DR offer the customer a choice of ISP-provided router/modem or ISP-provided modem only where customer can bring their own router, on the understanding that the ISP's responsibility stops at the LAN port of the modem and wireless is the customer's own problem - non-negotiable.
The modems are, of course, not just modems. They are routers in bridge mode. This is important, it permits operators to run diagnostics, right down to speed tests if the firmware is there.
Where the solution is all optical the ONT / media converter should be provided by the operator. There are plenty of options where a combined unit can do ONT and media converter duty depending on configuration which, IMHO, would be a good idea where possible. The operator can manage these and see light levels, etc, even across point to point fibre.
Ideally the operator should be in control of a managed device that delivers an Ethernet presentation to the customer's equipment. Whether that presentation includes full routing and wireless functionality should be down to individual choice, the default being that it does and 'modem mode' being an opt-in.