I have never thought capers tasted like anchovies, mind you they are used in tartar sauce so eaten with fish dishes as well as lamb so maybe thats why you thought that??.
I think they are more a sour olive tasting, unless they have been pickled then more vinegar tasting.
2 teaspoons ful of capers, 75-100g butter, 1 clove of garlic crushed, zest of 1 lemon, 1 tablespoon of lemon juice and a good pinch of parsley. heat butter then add everything else and for the last minute add in parsley.
there are a few recipes for caper sauces, this I use with fish but make a sauce with dijon mustard for lamb