How to say you don't want a gift back without implying that you DO ?

First of all, you are assuming that he will remember that the following day is your birthday. He may not. Just give the gift and make no mention of your birthday.