Swimming without warm ups is fine if you are doing it for a quick 39 minute session.
But if you are going to stay in for a marathon then warm up becomes a mandate. Because it helps your body to become loose for the rigorous swimming which might follow.
On alternate note, warm up is a good thing regardless. One more activity you can do along with friends having fun.
What’s stopping you?