The pipe extending from the drainage pipe on your old water heater is most likely a re-circulation line. The re-circulation line helps the water in the pipe to stay hot. It is not absolutely necessary, but it can reduce the amount of water that you waste before the water gets hot. The pipe may also serve as a pressure relief valve. This is the case if the pipe goes to a drain in the floor. The valve is an important safety feature of your water heater that will release water if the temperature and pressure inside your tank get too high. Without the valve, you run the risk of your tank exploding.