What is best for us to do?

Ohh wow. That's awful of him to say. Just give him some space, then he will see how much he misses you and how lonely his life is without you.