Try getting a sarong. It’s very light, comfortable and packs down to almost nothing; also it suits all purposes and occasions, including the temple etiquette.
You should take off your shoes and hat when entering a Buddhist or Hindu temple. If you’re carrying an umbrella, unfurl it. Your legs & shoulders should be covered; never enter a temple in beach wear (i.e. shorts or singlet).
It’s advisable to bring one set of Evening wear as larger hotels insist on ‘long attire’ for men during dinner (no shorts and no sandals). Ladies could come in dresses, skirts or long shorts, but not in any beach / swimwear.
Lightweight, light coloured cotton clothes are ideal for the warm climes of a tropical country like Sri Lanka. We’d like to recommend that you pack some warm clothes if you are planning to visit the hilly regions, where the evenings tend to get chilly.