That situation is very similar to how the backend treats its files. The design there is that when a file has registered that it wants events, the backend holds a counted ref to it so no other refs are necessary. I think that's reasonable since the file still is reachable - when the event comes it might very well run code that adds other refs to it again. When the file no longer is registered for any events, it effectively loses the ref from the backend and can disappear.
I suggest you use the same approach. If your events are single-shot, the callback you have would "deregister" its own object and thereby free a ref. If there are no other refs to it then, it'll get destructed at that point. Your exit callback should still "deregister" (and remove the extra ref) if the object is registered, to make it work if it's destructed explicitly.
This will make the async interface behave consistently with the file callbacks, which is a good thing imho.