Krems, along with neighboring cities Stein, Egelsee, Dürnstein and Spitz an der Donau are known for their world class wine production as well as their proximity to the Danube.
Use ÖBB's SCOTTY to look up train information
From the west take the regional train (Regionalzug, denoted by big R) from St. Pölten (usually every 30-50 min. every hour) to Krems.
All other directions regional (express) (Regionalexpress, denoted by a big REX) train from Vienna's Franz-Josefs-Bahnhof, Spittalau (U4, U6) or Heiligenstadt (U4) station which takes about one hour and leaves every hour.
From west: Highway A1 (Westautobahn) to Melk, B3 through Wachau. From South/East: Highway S33 from St. Pölten or A22/S5 from Vienna From North: B37 via Zwettl or through the Kamptal/Horn
There is a city bus system though with limited and serpentine routes. The city and its neighbors are all very accessible by foot and it's no more than a two hour walk from one end to the other.
Historic town centers of Krems and Stein. In particular:
There are a number of bike trails along the Danube, the town tourist bureau has numerious listings. Tour the monastery. Hike through the mountains and vineyards. Tour the winery.
The area is known for its wine which is available at reduced tariffs.
There are a number of restaurants in town serving the typical Austrian fare. Consider the local traditional eateries called "Heuriger", which offer inexpensive dinner and nice wines in a relaxed atmosphere.
As with the entire region there are a number of wonderful Heurigen (wine bars) in the area. The Piano in Stein offers beer from Austria, Germany and Czech Republic with a focus on beer from the near-by Waldviertel.
There's a number of guest houses in the area, such as the following.
For a comprehensive list see Krems - Tourism (german and english) 
Noteable places include:
Most of the Wachau (except Aggstein) can be reached by the local train Krems-Spitz or by bike on the Danube cycle track (Donauradweg). Göttweig is at the train station Krems-St.Pölten line and Melk at the Westbahn.
In summer many tourist boats operate in the Wachau. The biggest and most visible boat companies are DDSG (Donaudampschifffahrtsgesellschaft) and Brandner, but they are not necessarily the cheapest. MS Stadt Wien is a popular alternative among locals (weekends only).