Routing table size is going to be the issue if every one goes PI, although in time that may not become an issue with updated kit etc. The telecoms industry must have gone through something similar with number porting. You're correct as well, A&A will announce PI addressing.
However, devils advocate asks what actually fails to work currently with NAT, and would not work with the much more straightforward IPv6 NAT? I've come across a few cases where it's helpful for a front end server to have a native address, but those are individual border hosts not the end user clients. I'm not convinced that the requirement for those border gateways would go away without NAT because some of this is to do with opening firewall ports dynamically. (For example SIP early media needs protocol specific firewall support even without NAT).
I also note that in the recent past I've worked with two organisations that possessed Class A ranges and numbered their entire internal networks from those public address. Neither of them routed direct from end users to the Internet without NAT. I don't even know how it could work at that scale, in an organisation with probably thousands of separate Internet connections around the world.