It entirely depends on the sitauation in my opinion. Early game I guess 3-4 mins won't hurt. But later on, once people start to claim planets and gaining fleets, I suggest you pay close attention to your game. You need to organize your fleets and defence all the time. One weak spot can result is your downfall.
In order to reduce the possibility of syncs/crashes I suggest you try to organize and play a few games with trusted friends by LAN. This is how some SF play their games, and this way they have less crashes/syncs. If you want to play games by LAN (multiplayer -- online lobby like hw2 is Ironclad Online tab) get
Hamachi, create/join a network and play games by LAN.