Don’t become dependent on anyone’s company. If you go to swimming only with your buddies then change the narrative.
If hey don’t come then you will be tempted to not go. But resist that temptation and go solo. When you do this repeatedly you will have formed a habit where you are free to join or not join.
Remember if you find yourself not going on to swimming then better it be because you didn’t want to. And not because your friend won’t go.
Be independent or your choice to swimming. Sure it would be nice if your buddy tagged along. But be happy going solo swimming too.