When you take a day off from swimming, it is okay. But then when you don’t swim for a week, it starts to get serious. Because it is a part of your regime. An exercise routine – so your body reflects that.
If you don’t alter your diet then the lack of swimming will bring in more fat which ain’t nice.
Don’t let yourself get to a place where you are missing swimming. That means you had taken a long gap.
Instead take time off only till the time where you are anticipating the swimming. So no more time off than one week unless there is a real emergency.