Note: the freedom not to invite someone to speak at a student event because of their views is also part of the right to free speech. It is the same right used by the owners of the Daily Telegraph when they decide not to run a daily column by (eg) Owen Jones. https://twitter.com/christopherhope/status/1360721681774505990
You could argue that some platforms are so important that that they should be offered to speakers with a balanced range of views. But then you have to explain why that applies to a student society but not to the Daily Telegraph.
Universities (as opposed to student societies) may be a harder case: they are (essentially) public bodies. You can mount a good case for saying that they must offer a diverse range of views in teaching controversial topics.
But one of the issues with so-called “cancel culture” is that the standard examples trotted out usually involve institutions responding (frequently clumsily) to pressure from their customers, outraged at the expression of a view that they find unacceptable.
Universities are now - in accordance with Conservative policy - in a marketplace for students: and in a supplier/customer relationship with them.
Students/customers are entitled to say “I just don’t want to be taught by X” or “I don’t want to go to the university where X teaches”.
The problem with Conservatives complaining that universities shouldn’t respond to such pressures is that it cuts right across the market approach to university education that Conservatives have vigorously promoted for decades.
Quite how the free speech champion’s mandate negotiates its way through these thickets is going to be interesting to watch.
Incidentally, the observation that there is often a tension between the free market and diversity of opinion goes back (at least) to de Tocqueville.