Most people who follow a dairy-free diet are allergic to dairy products. However, a few people who are sensitive to lactose (milk sugar) also need to avoid all dairy products. Those who have reactions to lactose are called “lactose-intolerant.”