A few kilometres down the valley (back towards Akhaltsikhe) is another monastery up a steep track on the west side of the gorge. There is small community of friendly monks and a working chapel there as well as wonderful views of the local landscape. It is a great option to escape the larger crowds of tourist that tend to flock to the main monastery.
The site is accessible by bus.
There is a Georgian cab driver named Mamuka who lives in Tbilisi but his family home is in Aspindza just outside Vardzia. If you call him at 551086000 he will take you from Tbilisi to Vardzia and back for 200 lari (he speaks a little English, but better to communicate with him in Russian or Georgian if you're able). He is also generally willing to put people up at his home in Aspindza overnight if you don't want to make it the long day trip, and his mother is a good cook, although their house doesn't have indoor plumbing.
You can take public marshrutka from Tbilisi or Borjomi via Akhaltsikhe but you'll need to ask the driver where to drop you off. Most likely he'll take you to a guest house if you ask him and you are happy to go to his recommendation or trust whatever deal he has with the owner.