You may be thinking that most people don't know about Yelp, or care about Yelp, so why worry about it? Your right. But understand that lots of people DO know about Yelp and lots of people trust Yelp. And Yelp is growing.
Sure, most people don't know or care about Yelp. That's true. However, if people learn that your business has a low average rating, or lots of negative reviews, then word can spread. And this is especially true if your business is for a minority or a small community. So even if people don't know about Yelp, the word can spread about your buisness having a low average or bad reviews.
And actually, when a person DOESN'T know about Yelp that can be bad. Sometimes if a person knows how Yelp works, then they may not trust it. But if a person DOES NOT know how Yelp works, then they don't know how it works. So all they know is that your buesiness listing has a low average rating or bad reviews. So they naturally might be hesitant about wanting to do business with you.
Sure, you may think that if people don't know about Yelp then how can they find you on Yelp? Easy. Search engine results.