Yes, you need to do the thing on continuous basis for the growth. And adopting a daily habit helps a lot. But when you do something physical, it is alright to take a rest for one day. Given you rejoin the next day.
Our body works in mysterious way, so when you take a planned periodic rest – it helps your muscle memory to improve and you will able to do the thing much faster.
For example, if you think about swimming – the ability to swim without focusing on your hands and legs is when you have learnt. This only happens when the task is stored in your muscle memory. Also walking, learning to do handstand or walking down home from school. All this happens because of muscle memory.
So if you need to level up in the physical training, you need to let those muscle memory develop.