The valve on the side of the hot water tank is called a PRV or pressure relief valve. This valve is designed to open if the pressure in the tank gets too high. If the PRV was not in place the tank can explode with as much force as a bomb. So needless to say the valve must be there. Because of the spring loaded design of this valve it has a tendency to leak. The good news is that the valve can be replaced. Make sure the water tank is shut off and cool, the water supply is shut down and there is no pressure on the valve. I personally have not changed one of these so I can’t give any particular tips other than to hire a plumber if you are unsure how to procede.