Legendary gems can be found in the gem mine of the fortress as soon as there is at least one picture of a Legendary item in the library. The more Legendary items* you have unlocked in the library, the higher the chance of finding a Legendary gem in the gem mine.

You can also get legendary gems as final reward during the event Summer Collectifun.

* Legendary items are rewarded for defeating the boss in Legendary dungeons.