Food is not the reason animals get worms, they have to actually pick up the parasite from their environment. You shouldn't give milk for other reasons like intestinal upset and calcium imbalance. If he didn't get tummy upset, or didn't get it that often, don't worry about it. If he developes diarrehea, just stop letting him have the milk. Is his name Scouba?