I think you already know you’re doing the right thing. It doesn’t sound like there’s much to love about him except that he’s a fellow human being. He certainly didn’t act in a way to deserve your love.
Unfortunately, we (especially women) tend to be like a moth to a flame. Just know that you’re not alone in this kind of situation.
Also, your love for your child will far exceed any love you might have for this man. You’ll want to protect a baby even more than yourself, and since you’re smart enough to do what’s right for you, it’ll be easier to continue doing so for a child. Surround yourself with people who support your decision so they can keep you grounded when you start to be tempted by him.