A sexual meeting should only occur when two people choose each other and both want sex at the same time.
Your problem is your belief that it is dangerous to disappoint him. But all of us get disappointed many times in life. Disappointment is a natural feeling when you do not get what you want. It is not a happy feeling, but it is definitely not dangerous, except if you suppress it.
Accept his disappointment, and accept that he may choose to satisfy himself, if you do not feel like having sex at a particular time.
Be considerate, do not torment him by arousing him when you do not yourself feel like having sex.