Alicia Silverstone, Olivia Wilde, and Lea Michele are just a few celebs who embrace the vegan lifestyle. And while they may tout the beauty benefits of this eating plan, the reality is a little more complicated.

“You need to listen to your body,” says Frank Lipman, MD, founder of Eleven-Eleven Wellness Center in New York City. “Some people do really well on a vegetarian diet, but some people just aren’t meant to be vegetarians. If you’re not feeling vital every morning — your skin should look good, you shouldn’t have bloating and fatigue — then whatever diet you’re on is probably not for you.”