Look at it from an individual business' point of view. International trade has challenges. You have to deal with different rules and regulations, custom checks, tariffs and who knows what else.
This is where the idea of a single market comes in. It expands what "within our country" means. It makes it easy to forget that Finland, France, Germany, Slovakia, Romania are different countries. From an individual business' point of view, it makes no difference whether you are buying from or selling to Helsinki, Frankfurt or Milano.
That, my friend, is the exact opposite of "not allowing free trade".
Especially considering that those 450 million cover most of the richer part of the world.
What is worse, the single market made trading so easy that some people, particularly in the country that has been subject to decades of relentless anti-EU propaganda, forgot that it was the single market that made it easy. They take their luck and privilege for granted and are whinging when they foolishly throw it away and suddenly see the consequences. Like, for example, Brexit voting British fishermen.