Doctors advise avoiding milk if you have these health issues