Take one of the many regular buses to Trat from either the Eastern Bus Terminal (Ekamai) or the Northern Bus Terminal (Moh Chit). Direct 1st class buses from Ekamai average a bit more than 5 hours, the less frequent frequent services from Moh Chit take a little less. While most 2nd class buses take a lot longer as they make more stops and use slower and less direct routes, the midnight departure from Ekamai takes only 4.5 hours.
From Trat, take a minibus direct to the Hat Lek border crossing - departures every 45 minutes between 06:00 and 18:00, takes between 1.5 and 2 hours, 110 baht.
Cross the border to Koh Kong but be aware that the border opens at 07:00 and closes at 20:00. If you want to go on directly to Sihanoukville by sea, keep in mind that the boat is scheduled to leave at 08:00 (in practice it often leaves a little late - typically around 08:20).