The Caucasus region is a mountain range lying between the Black Sea and the Caspian Sea, considered part of the natural boundary between Europe and Asia. Geographically it is usually considered part of Western Asia, adjacent to northeastern Turkey and northwestern Iran. But culturally, this portion of Russia and these small former Soviet republics are arguably part of Eastern Europe suffering from the same ethnic-hatred and tension that has plagued the Balkans including recent wars in Georgia, Azerbaijan and Chechnya.
The countries and territories of the Caucasus are all isolated but ancient lands inhabited by what is likely the world’s most ethnically diverse region. All of places mentioned here were annexed by the Soviet Union at some point, only to gain independence in the 1990s. Unfortunately since then the area has witnessed several ethnic conflicts, civil wars, and other conflicts both between and inside states. Several regions such as Abkhazia, South Ossetia, and Nagorno-Karabakh gained virtual independence this way, but few nations recognize the legitimacy of these places. While fighting continues in Russia's North Caucasus, countries here should be fairly safe.
While traveling here expect to meet friendly locals, eat food like none other on earth, and witness breathtaking mountain vistas. But also expect expensive and bureaucratic visa procedures, crooked police and all the other blights. Armenia and Georgia are suffering from economic recession due to an economic blockade of Armenia by Azerbaijan and Turkey and of Georgia by Russia, and oil-rich Azerbaijan is essentially an old-fashioned dictatorship where the oil wealth is still not flowing to the entire population but to a small elite. The old ways of the Soviet Union are never far away when dealing with any kind of official person including immigration officers and police. And be very, very careful before you enter an area which is controlled by the Russian backed local militia or Russian border troops of the separatist regions in Georgia and by the Armenian backed local militia of the separatist region in Azerbaijan.
The Caucasus is one of the most complex linguistic regions in the world, containing more than 60 languages from five distinct language families. This linguistic diversity in and of itself is a major draw for anyone interested in linguistics, but it also lends the region one of its most alluring charms - cultural diversity.
Since the end of the Soviet Union, the Caucasus has become decidedly less cosmopolitan as ethnic groups have migrated to their "heritage" countries. This ethnolinguistic segregation has been especially deep where there has been ethnic conflict, such as between Armenians and Azeris, Abkhaz and Georgians, and Ossetes and Georgians. Because of this trend, there is less inter-ethnic interaction and therefore people are less multilingual than in the past. National languages are becoming ever more important to travelers in the region as fewer locals understand languages other than their own. Thus, a traveler to Georgia would benefit from Georgian, a traveler to Azerbaijan - Azeri, a traveler to Armenia - Armenian, etc.
Russian remains the lingua franca of the former Soviet nations of the Caucasus and the most useful language for any traveler intent on visiting multiple countries in the Caucasus. The current trend is for English language study to displace Russian, but the spread of English proficiency remains extremely limited in all four countries of the Caucasus. As a rule, older people are more likely to speak Russian while younger people are more likely to speak a little English or no foreign language at all. Similarly, citizens who are ethnic minorities within their country are more likely to speak Russian because it is a means of inter-ethnic communication. Travelers can expect that ethnolinguistic minorities within Russia, Abkhazia, or South Ossetia will speak Russian, except in very small, isolated villages.
Border crossing is generally difficult throughout the Caucasus. Both the Russian-Georgian border near Kazbegi and the Russia-Azerbaijan border are only open for citizens of CIS countries. For non-CIS citizens, there is no way of entering/exiting Russia through the Caucasus. Aside from flying, there are ferries between Sochi, Russia & Trabzon, Turkey (near Georgia) and Baku, Azerbaijan & Aktau, Kazakhstan (near Russia).
The Armenian-Azerbaijani border is closed because both countries remain at war. The Armenian-Turkish border is also closed due to tensions between both countries. To travel overland between Armenia and Azerbaijan and Armenia and Turkey, it is necessary to go through either Georgia or Iran.
Georgia's borders with Turkey, Armenia, and Azerbaijan are all open, making the country somewhat of a regional transit hub for the Caucasus. Since 2003's Rose Revolution in Georgia, bribes are absolutely not necessary for foreign travelers entering Georgia. However this cannot be guaranteed for Armenia and Azerbaijan.
Entering Azerbaijan with a used Armenian visa or vice versa could likely cause problems (suspicion) with border guards, but shouldn't prohibit entry. Nevertheless, it is recommended to visit Azerbaijan first and then Armenia, to avoid potential problems and a refusal of entry to Azerbaijan. However, you will not be allowed entry to Azerbaijan with a Nagorno-Karabakh visa (you can ask to get the NKR visa on a separate piece of paper, though), otherwise it would result with a permanent ban of entry to Azerbaijan.
Naxchivan (Azerbaijan) can be entered from Turkey and Iran.
Overnight trains travel between Tbilisi-Yerevan and Tbilisi-Baku. When traveling by rail, you have the option of rooms containing 4 beds (coupe, pronounced koo-peh') or 2 beds (SV, pronounced es veh). SV is a bit more expensive, but more comfortable and generally considered more safe from pickpockets.
There are direct bus services between Tbilisi-Yerevan and Tbilisi-Baku. If taking the air-conditioned bus between Tbilisi-Baku, bring a jacket! Buses also operate across the Russian-Georgian border, but are not an option for non-CIS country nationals.
UPDATE (April, 2012): Minbuses operate between Baku and Sheki (6 hr), Lankaran (5 hr), Xachmaz (3 hr), Saatli (5 hr), and Imishli (4 hr). A minibus also runs from the Georgian border at Krazny Most (Red Bridge) and should cost about 10 or 12 mannat. It can be picked up at either side of the border (don't worry if they ask you to pay on the Georgian side - they turn up to pick you up. Insist on bringing your own bags across, however). Journey time to Baku should take about 8 hours. Be warned, driving in Azerbaijan is a genuinely scary experience. Virtually all drivers have scant regard for the rules of the road and the standard of roads themselves is shockingly poor. It is emphatically not for the faint-hearted, so whilst the long train might challenge your stamina but not your nerves. Check AZAL flights from Tbilis-Bake well in advance for some reasonable deals.
Return to Tbilisi can be caught at the indescribably chaotic bus station, which doubles up as an eerily quiet shooping-centre (take bus 65 from outside Double Gate in the old town for the 20min trip, which leaves you with a 400m long dodgy stretch of road/highway with pestering taxi-drivers on which to walk: Cost 20q) or simply taxi it from the centre for approx. 15 mannat (worth it!), which saves on hassle. Both buses and minibuses are available from this station direct to Tbilisi, price approx 12 mannat for both. Bus counter 26 at very back of ground floor. The bus is a few hours slower and not guaranteed to pick you up once you cross the Georgian border, so minibus is preferable.
If you would prefer a more social mode of transport, minivans (marshrutkas) operate across all open borders and throughout the entire Caucasus region.
There are direct flights between Tbilisi to Baku, Tbilisi to Yerevan. Expect no trouble at the airports--they are small and efficient.
Car rental is more expensive in the Caucasus than in the West, but car hire with a driver is quite affordable. For international travel, however, it will be necessary to pay for your driver's lodging unless he was already planning to make the trip.
Ski in two very beautiful ski resorts of Georgia. Bakuriani and Gudauri.
Relaxing on the Georgian beaches on the black sea in Batumi, Kobuleti, Ureki, Gonia etc.
The Georgian "Khinkali" and "Khachapuri"!
The drinks of note in the Caucasus are Georgian wines, Armenian cognac (brandy), and Russian vodkas. Local beers throughout the Caucasus are excellent values.
Especially tasty Georgian wines:
The Caucasus is a tinderbox of age-old rivalries, some frozen, some very hot indeed. Much of the Russian North Caucasus is an active war zone, and the rest suffers from extremely high crime rates. Fragile ceasefires are more or less holding in Georgia's disputed regions of South Ossetia and Abkhazia (which are both quite dangerous), as well as the border between Azerbaijan and Armenia, still officially at war over Nagorno-Karabakh.
However safety is not a serious issue to worry in Armenia and Azerbaijan if you are not too close to the border between the two countries. It is safe to visit even Nagorno-Karabakh from the Armenian side. Georgia is trickier with regards to personal safety, with higher violent crime and carjacking rates in some of the bigger cities, and occasional kidnappings in some of the wilder off the beaten path locales.