Libertarians (or whatever you call yourselves) have to come to terms with the fact that libertarian ideology only describes your basic obligations in your interactions with others. You need something else to direct you towards the "good life." Otherwise you'll lead a shallow...
...existence of moral agnosticism. That heroin addict might not be infringing on your rights, but do you want to live in a community where his behavior is acceptable? Would he be a good neighbor? Would you let him watch your kids if you stepped away from the house for a bit?
Those getting hung in my heroin addict example need to take your autism pills.