It is also believed that long ago Guru Padmasambhava meditated in the caves around Rizong years before the monasteries were built.[6] It is also inferred that in the small caves in the vicinity, Lamas used to meditate for years in isolation from the rest of the villages. They subsisted on one meal a day, which was provided to them by local people through a 1 foot (0.30 m) square window opening in the cave.

Before the monastery was built in 1831, it was started as a hermitage for teaching the Buddhist religion to the monks, with a strict regimen of a celibate life suited to the monastic order.

In the 18th century, Lama Tsultim Nima who meditated at the rDzong-lung mountains decided to establish a hermitage (before he built the present large monastery) here, as a monastery for monks to meditate and learn the teachings of Buddha. Supported by many monks, initially many mud huts were built where they recited gso-shyong. He laid down very strict rules of celibacy called the “Vinaya Rules” to be followed by each monk who meditated here. In brief, rules set are the following.
Monks are not allowed to leave the monastery, except in the case of sickness
No comforts of bedding are allowed to sleep at night
Monks are not to touch anything handled by women (including their own or others sisters)
Before sun rise or after sunset, Monks can not leave their cell, except to bring water
Not even a needle worth of possessions are allowed to be owned by the monks
Fire cannot be lit in their rooms
Any kind of donation received by a monk from his home shall be shared with other monks in the hermitage
The boundary of the hermitage was marked by three types of fences and no women was allowed to sleep even in the outer most boundary of the hermitage
Any rumour about offences that the monks committed would result in their rustication from the monastery
Within the ambit of the above rules, the monks of the monastery would at times become quite sentimental about even inadvertently treading on an insect or even cutting a blade of grass. Over the years, the hermitage became a place of worship and pilgrimage to all Buddhists from Ladakh. It is reported that the king of Ladakh gave rich donations to convert the Hermitage into a retreat centre and the queen of Ladakh even visited this place on a pilgrimage. At this stage, as the number of monks in the hermitage increased, Lama Tsultim Nima decided to build a much larger monastery due to the then location of the hermitage being inadequate to build one large monastery.