Most bus companies stop at the Erzurum Otogar, which is about 1km away from the main city centre. It is a walkable distance to town, or you can take the public bus to/from the Otogar.
Erzurum is a pretty small city, and you can cover most of it by foot, unless you are thinking of heading out and around the city, in which case, most public buses and taxis will be able to get you there.
Citadel of Erzurum
The Citadel of Erzurum is situated on the top of the hill in the middle of the city centre. The YTL 2 admission fee allows you to explore the grounds as well as to climb up the minaret/clock tower, as well as visit the citadel masjid. It's worth the effort to climb up the clock tower as it gives you a commanding view of the city as well as the surrounding mountains.