Use the blacksmith's "Extract gem" service to remove gems from items.

The removed gem lands in your inventory. Unfortunately, the item in which the gem was socketed is destroyed without replacement. However, you will receive a proportionate amount of resources back if this item was previously upgraded by the blacksmith. This does not apply to e.g. weapons that were thrown into the toilet as they lose their sales value there.

To extract a gem, drag an item onto the corresponding area at the blacksmith or, if the item is not equipped, use the right-click context menu on the item whose gem you want to extract.

Extracting a gem costs metal and arcane splinters. Alternatively, you can also spend 10 mushrooms.