Ban Phe exists as a port for ferries to Ko Samet and for limited fishing. Ferries between Ban Phe and Ko Samet usually take 20-45 minutes, depending on the speed of the boat and which beach/pier they use.
There are direct 1st class air-con buses between Ekamai (Eastern) Bus Terminal in Bangkok and Ban Phe, departing hourly.
Turn right off Sukhumvit Road at kilometre 231 and keep going for 5 kilometres or turn right at kilometre 238 and keep going for 3 kilometres.
Bangkok Airport Limousines  will drive you to the pier in 2.5 hours or less in a luxury sedan. Taxi's will want to negotiate a rate before accepting the ride, because they run a risk of coming back empty.
Ban Phe is little town and not too difficult to traverse. The entire town is about 4 km long so walking is possible. There are also plenty of motorcycle taxis.
Going to Koh Samet you can take a speed boat for more than 2000 THB or take the ferry for 50 THB, Arrived you will take taxi and have to pay entrance fee to national nature park.
Want to see some things then ask the staff at Bedrock, Christies or Tawan-Inn for their ideas they will help you. Both Tawan-Inn and Bedrock got motorcycles for rent.
While the beaches in the immediate area of Ban Phe are not very nice due to the pier, go a few km in either direction and the beaches are quite nice and practically devoid of western tourists.
Heading east out of town 2 km leads you to Suan Son National Pine Forest. The road runs right along the beach and the farther away from town you get the nicer the water is. Once you get 3 km from town the beaches are almost as nice as those on Samet. With only one short incursion inland, the road basically runs along the sea all the way to Lam Mei Pim, 24 km away. There are several five-star resorts and condos along this stretch of beach
Heading west will eventually lead you to the beach Hat Mei Rum Pung. The road is not direct, but if you make the trip you will find 7 km of very nice beach with a number of restaurants, resorts and cheap hotels.
In the fish market, many shops offer local seafood products such as fish sauce, sun-dried cuttlefish, dried shrimp, etc. A half mile from the fish market are the local tourist trinket markets -- key chains made from sea shells and grotesque blowfish lamps. Most of this tchatchke is meant for the native tourists who stream down from Bangkok on weekends. But if you are a serious sea shell collector you can find some incredible bargains tucked away at the stalls in the back; huge conch shells and glittering cowries for just a few dollars USD.
Bedrock Bar Grill Quality western food and they have seperate kitchen for traditional thai food. The bar is unique, done up like the flintsones house (hence the name Bedrock) and great music. prices are very competitive. Christies Bar is a quality venue, great bar, staff, food (western and thai) and music. Tawan-Inn: Has a chef with 10 years experience in Europe, who prepares both Thai and Western food of high quality. Dining outside in the garden is possible.
A few great bars one being Bedrock Bar and the other being Christies they are about 350 metres on both sides of Tawan-Inn. Beer is very cold everywhere and they are all stocked with wines and spirits. Prices don't differ too much, though Christies is most expensive.Tawan-Inn has a nice garden to sit in.