You are not subsidizing anyone at all. Your taxes do not depend on fiscal outlays, especially relatively small ones like this. That fact has been established over and over again.
And people not going to college are not paying for anyone's student loans. They don't pay much tax period.
It takes a much lower person to wish vengeance upon people for whom college didn't work out that well. A lot of people with big student loan bills are people who took out money and couldn't finish school for one reason or another -- whether it was poor performance or personal tragedy. If they started a business and failed, they could wipe out their debts in bankruptcy. Not so with student loans.
I had a student whose brother was a competitive skier and who died in a freak skiing accident. She withdrew from school in April because she was too devastated to take her finals. She did not return. 3 semesters of law school. I had another student who withdrew when her husband came back from Afghanistan with a traumatic brain injury. She was in her 5th semester. She did not return. I had another student who got cancer. She actually stayed in school for three semesters while receiving chemo.
But hey, who gives a shit about those people.
That last student had one of the saddest stories I've ever heard. She was a Roma woman who was raped by Serbian soldiers as a young teen. She got pregnant, and because of her traditional culture, she was shunned by much of her family and abortion would have meant exile. So she resettled in the US. Meanwhile, she developed severe epilepsy and had a rescue dog accompany her everywhere. Then she got cancer. Died about a year after she would have graduated.
And the reason I know a lot about her was that she confided in me because some of the other professors were super-assholish to her. There were professors who were really unhappy about the prospect of having a rescue dog in class. They didn't exactly tell her she couldn't, but she made it uncomfortable enough that she had to change sections multiple times. There were like 5-8 law professors she trusted. It kind of sucked for her, because she hated corporate law, but she took my classes anyway because she knew I would always be an ally.