Of course.
They may decide not to apply the same visa exemptions they do to other developed and middle income countries such as the USA, Brazil, Canada, South Korea, Mexico, Japan, Colombia etc and make us get visas in advance as they do for citizens of India, Sudan, Russia and other countries perceived to be an overstay risk.
https://en.wikipedia.org/wiki/Visa_p...isa_exemptions
It's their choice, that's what sovereignty (in this case pooled) looks like. And who would want to go somewhere, against the will of the receiving nation, but as part of some "right" they were enforcing against it?
But that citizens of British Overseas Territories who are not EU citizens are on the same list as the Americans rather than the Nigerians may be a clue as to which way they'll go on that.