You may not be able to walk between the beach and the caves - too far and too steep. The walk between Caves House and the Caves is a nice bush walk. A car or a tour are the best options to see the area.
Ngilgi Caves (the first 'g' is silent) are just out of town on the Dunsborough road. The caves have a long history of being show caves, attracting tourists from Perth for over 100 years. The tours are semi-guided, taking you down in the caves, and giving you as much time as you like to wonder around and see the formations. You can just lie on the ground and look up at the formations from the roof of the cavern for as long as you are inclined. The kids will like the crawl tunnel. A joint ticket is available with the Cape Naturaliste Lighthouse.
Snorkelling. Yallingup has the southernmost occurrence of tropical fish in Western Australia, and there is a protected reef just to the south of the main surfing beach. Take your snorkel and have a look around.
Swimming or Surfing The beach is sandy, with a reef break further out.