For my Filson vest, I used camo moleskin that is adhesive backed that they sell in the archery section of most sporting goods store. It's used to soundproof the rest area on a bow if an arrow slips off of the rest. I just wrapped it around the buckles.

You can see it in this picture. You could also get green and/or black adhesive backed felt at any craft store for less. I like the velcro/leather idea. I was thinking about a more permanent solution but it has been like this for about 5 years.

[Linked Image from live.staticflickr.com]