Should I ask him out?

Go ahead because you'll never know if you don't. If he doesn't like you then move on but if it ruins the friendship then you guys were never to be together. I know it sounds a bit hurtful but it has happen to me. If you really don't want to take a gamble then really make sure by asking some of his friends that if he likes you.