The island in the back of the second picture above is called Wu Chau and the one in the picture immediately below with the bump on the right is to its north and is called Sam Pui Chau. It's quite a distinctive shape as you can see.
The wooden pier used in the film looks like it still exists - minus the Chinese style embellishment - and looks as though it is now private and used by a nearby organic farm. Sadly, there is no Streetview for this particular neck of the woods at the moment.
Furthermore, there is a brief scene where Penn's character finds himself stranded in a remote area and flags down and convenient passing taxi. Although it has drastically changed with village house development (the Tseng Tau village mentioned in the title seems to have expanded considerably since filming was completed), there is still a small strip of road leading out of the village and it looks like this is where the following scene was filmed - with Three Fathoms Cove still in the background.