Delete him from your phone but give a good friend his number in case you ever need it for some reason (who knows what that could be - you left something of value at his place and would like him to return it? you find something of his? who knows) but instruct them not to give it to you when you've been drinking
I'm sorry you're going through this. Sounds like it has nothing to do with you and everything to do with him. If it was about you, he would have told you.