The Caucasus region is a mountain range lying between the Black Sea and the Caspian Sea, considered part of the natural boundary between Europe and Asia. Geographically it is usually considered part of Western Asia, adjacent to northeastern Turkey and northwestern Iran. But culturally, this portion of Russia and these small former Soviet republics are also part of Eastern Europe suffering from the same ethnic-nationalism and tension that has plagued the Balkans including recent wars in Georgia, Azerbaijan and Chechnya.
The countries and territories of the Caucasus are all isolated but ancient lands inhabited by what is likely the world’s most ethnically diverse region. All of places mentioned here were annexed by the Soviet Union at some point, only to gain independence in the 1990s. Unfortunately since then the area has witnessed several ethnic conflicts, civil wars, and other conflicts both between and inside states. Several regions such as Abkhazia, South Ossetia, and Nagorno-Karabakh gained virtual independence this way, but few nations recognize the legitimacy of these places. While fighting continues in Russia's North Caucasus, countries here should be fairly safe.
While traveling here expect to meet friendly locals, eat food like none other on earth, and witness breathtaking mountain vistas. But also expect expensive and bureaucratic visa procedures, crooked police and all the other blights. Armenia and Georgia are suffering from economic recession due to an economic blockade of Armenia by Azerbaijan and Turkey and of Georgia by Russia, and oil-rich Azerbaijan is essentially an old-fashioned dictatorship where the oil wealth is still not flowing to the entire population but to a small elite. The old ways of the Soviet Union are never far away when dealing with any kind of official person including immigration officers and police. And be very, very careful before you enter an area which is controlled by the Russian backed local militia or Russian border troops of the separatist regions in Georgia and by the Armenian backed local militia of the separatist region in Azerbaijan.
The Caucasus is one of the most complex linguistic regions in the world, containing more than 60 languages from five distinct language families. This linguistic diversity in and of itself is a major draw for anyone interested in linguistics, but it also lends the region one of its most alluring charms - cultural diversity.
Since the end of the Soviet Union, the Caucasus has become decidedly less cosmopolitan as ethnic groups have migrated to their "heritage" countries. This ethnolinguistic segregation has been especially deep where there has been ethnic conflict, such as between Armenians and Azeris, Abkhaz and Georgians, and Ossetes and Georgians. Because of this trend, there is less inter-ethnic interaction and therefore people are less multilingual than in the past. National languages are becoming ever more important to travelers in the region as fewer locals understand languages other than their own. Thus, a traveler to Georgia would benefit from Georgian, a traveler to Azerbaijan - Azeri, a traveler to Armenia - Armenian, etc.
Russian remains the lingua franca of the former Soviet nations of the Caucasus and the most useful language for any traveler intent on visiting multiple countries in the Caucasus. The current trend is for English language study to displace Russian, but the spread of English proficiency remains extremely limited in all four countries of the Caucasus. As a rule, older people are more likely to speak Russian while younger people are more likely to speak a little English or no foreign language at all. Similarly, citizens who are ethnic minorities within their country are more likely to speak Russian because it is a means of inter-ethnic communication. Travelers can expect that ethnolinguistic minorities within Russia, Abkhazia, or South Ossetia will speak Russian, except in very small, isolated villages.
Border crossing is generally difficult throughout the Caucasus. Both the Russian-Georgian border near Kazbegi (open since 2012) and the Russia-Azerbaijan border are only open for citizens of CIS countries. For non-CIS citizens, there is no way of entering/exiting Russia through the Caucasus.
UPDATE: THIS IS NO LONGER THE CASE; both the border crossing at Verkhniy Lars (near Kazbegi, using the Georgian Military Road between Tbilisi and Vladikavkaz) and the border crossing between Azerbaijan and Dagestan, have been reopened for non-CIS/ex-Soviet citizens, providing at least two land based entry/exit points to/from Russia.
The Armenian-Azerbaijani border is closed because both countries remain at war. The Armenian-Turkish border is also closed due to tensions between both countries. To travel overland between Armenia and Azerbaijan and Armenia and Turkey, it is necessary to go through either Georgia or Iran.
Georgia's borders with Turkey, Armenia, and Azerbaijan are all open, making the country somewhat of a regional transit hub for the Caucasus. Since 2003's Rose Revolution in Georgia, bribes are absolutely not necessary for foreign travelers entering Georgia. However this cannot be guaranteed for Armenia and Azerbaijan. Speaking from personal experience though, there's no reason to believe bribes are an issue in Armenia. Having been there twice, this was NOT the case. Indeed, the police and officials behaviour seems pretty much as in Georgia (thus, trustworthy and non-corrupt).
Entering Azerbaijan with a used Armenian visa or vice versa could likely cause problems (suspicion) with border guards, but shouldn't prohibit entry. Nevertheless, it is recommended to visit Azerbaijan first and then Armenia, to avoid potential problems and a refusal of entry to Azerbaijan. However, you will not be allowed entry to Azerbaijan with a Nagorno-Karabakh visa (you can ask to get the NKR visa on a separate piece of paper, though), otherwise it would result with a permanent ban of entry to Azerbaijan.
Naxchivan (Azerbaijan) can be entered from Turkey and Iran.
Overnight trains travel between Tbilisi-Yerevan and Tbilisi-Baku. When traveling by rail, you have the option of rooms containing 4 beds (coupe, pronounced koo-peh') or 2 beds (SV, pronounced es veh). SV is a bit more expensive, but more comfortable and generally considered more safe from pickpockets.
There are direct bus services between Tbilisi-Yerevan and Tbilisi-Baku. If taking the air-conditioned bus between Tbilisi-Baku, bring a jacket! Buses also operate across the Russian-Georgian border, but are not an option for non-CIS country nationals.
If you would prefer a more social mode of transport, minivans (marshrutkas) operate across all open borders and throughout the entire Caucasus region.
There are direct flights between Tbilisi to Baku, Tbilisi to Yerevan. Expect no trouble at the airports--they are small and efficient.
Car rental is more expensive in the Caucasus than in the West, but car hire with a driver is quite affordable. For international travel, however, it will be necessary to pay for your driver's lodging unless he was already planning to make the trip.
Ski in two very beautiful ski resorts of Georgia. Bakuriani and Gudauri.
Relaxing on the Georgian beaches on the black sea in Batumi, Kobuleti, Ureki, Gonia etc.
The Georgian "Khinkali" and "Khachapuri"!
The Azerbaijanian "Yarpaq dolmasi, goy qutab (green qutab), et qutabi (qutab with meat) etc." If you love meat, eat "Kabab, dushbara" (dushbara it's good to eat in cold weather).
The drinks of note in the Caucasus are Georgian wines, Armenian cognac (brandy), and Russian vodkas. Local beers throughout the Caucasus are excellent values.
Especially tasty Georgian wines:
The Caucasus is a tinderbox of age-old rivalries, some frozen, some very hot indeed. Much of the Russian North Caucasus is an active war zone, and the rest suffers from extremely high crime rates. Fragile ceasefires are more or less holding in Georgia's disputed regions of South Ossetia and Abkhazia (which are both quite dangerous), as well as the border between Azerbaijan and Armenia, still officially at war over Nagorno-Karabakh.
However safety is not a serious issue to worry in Armenia and Azerbaijan if you are not too close to the border between the two countries. It is safe to visit even Nagorno-Karabakh from the Armenian side. But it's illegal, this therritory is not recognised by any country, not even by Armenia.
Outside of the areas close to the disputed territories of Abkhazia and South Ossetia, where the risk of violent crime and kidnapping is significantly higher, Georgia has one of the lowest crime rates in Europe. Georgians are a very hospitable people. That having been said, visitors are advised not to talk about Russia or the disputed territories. Hate against Russians is extremely high amongst Georgians, to the extent that such discussions may incite violence.
Chechen, Georgian, Armenian and Azerbaijani mafia groups are considered to be notorious in the former Soviet Union but are mostly centered in the Caucasian expat communities and are far from noticeable in their home countries.