Should I take my son away from his mother and her lover?

If you have proof of them refusing to feed him, that is neglect and needs to be reported to CPS. However, just because your ex is a doesn't mean that you have any more of a right to your child than she does. Just keep that in mind, in case this goes to court for custody, that without a real reason for requested custody except "I don't think my son should be hanging out with a couple," you might not get much slack in court.