You should shut it off. But if you start going and get a few blocks and start stalling, don't start cursing your bike. Just remember to put petcock back on. Did that 3-4 x before I got in the habit of returning petcock to on when starting.
You are right on there, if I remember to shut it off, I'll surely forget to turn it back on! I didn't even shut if off over the winter, plan to do it from now on
though.