According to Boston plumber Basnett Plumbing, Heating, and Air Conditioning, if you hear a rumbling sound coming from your water heater, there could be a few reasons for it. One reason that they cite is hard water scale. Another reason is a buildup of sediment. Both scale and sediment can cause the water heater to overheat. An excessive amount of cold water entering the water heater tank could also cause the rumbling sound.