I like the idea, but, the reason I'm not sure about doing so is because the gems are embedded into the skin of each race of characters. These gems also allow each person born with them to use magic.
Maybe there are other crystals that people use as jewelry. I'll play with the idea and post them here later this week for more feedback. I'll try to work the idea in.