A long time ago, when I was in law school, my criminal law professor addressed the exact question you posed. He said as a starting point that the American system of freedom and justice was dependent on everyone having legal representation when charged with a crime. Not just the chosen few who the legal elite deemed worthy of representation, but everyone.
However, he added, IIRC, "You have to have personal boundaries that are narrow enough so some defendants are not denied representation." He gave himself, who did pro bono work, as an example of the type of absurd thinking that goes into such line drawing. He said that he had once represented, pro bono, a man accused of murdering and raping a woman. He said it wasn't until he had confirmed that the rape occurred after the murder that he agreed to represent the suspect. He said the distinction he drew in that case was absurd and meaningless, but he stressed that these sort of meaningless distinctions were important for maintaining any sort self-respect, which he deemed particularly important for the legal profession. In his case, he refused to represent anyone who had raped a woman while she was alive. He added that he knew there were lawyers who would represent men accused of sexually assaulting women, so his refusal to do so was not denying such men legal counsel.