"You" is a weakness

If I were addressing a group and said, "I need you to come with me." ,the people in the group would not know if I was addressing just one of them, or the whole group."

This is a silly idea. You're addressing the group and not an individual in the group. It's clear that you are addressing the group until you address the individual. (or is this about learning the concept as a non-native English speaker?)