What
@COYBL25 says. If you postpone cancel it; the next time he'll ask the anxiety will be more severe more prominent. You have to push through because they are like an ink spot; they can quickly expand and start manifesting in other areas/aspects of your life. After I had my first one suddenly I thought maybe I'll also have it there. Needless to say that did happen. You just have to do it. The first time is hard, the second time is also a bit hard. The third time (can also be the fourth/fifth just an example here) I started to enjoy myself again. It's easier said than done of course but it's the way to go.
Also I've noticed that most anxiety attacks follow this pattern:
So for some activities, say you're stressed out because you have to give a speech, you can go sooner (to the venue/office) so you peak earlier get the anxiety attack over with and do whatever you have to do. Mainly helpful for anticipation anxiety. I don't know how your anxiety attacks manifest but with me it's /tense throat/ heart palpitation/vomiting (not the end of the world, just not good for the teeth so special toothpaste is required to avoid teeth erosion). After the vomiting I always feel better. Or derealisation, tbf that's just bloody annoying. My trick is counting your limbs or such. All in all they are not very frequent you just have to know how to deal with them. Most people don't even notice when I have them.
It would be good for you to start doing things outside your routine, it really helps. Again the first, second time... it can be annoying but gradually you'll start enjoying yourself and you'll get more confidence with other things that exist outside your routine. It's a gradual process though, but you'll reap benefits.
Shouldn't feel bad about it.