Secret Santa assessment

I’m posting this half-baked idea here because (i) I think it’s neat, (ii) I love the cute name and want to share it, and (iii) it’s likely a little too far outside my area of expertise and will take too much effort for me to (credibly) get through as peer review.

So let’s beta-test it. Is it interesting or useless? Is it novel or has it already been proposed? Let me know!

Introduction

Misaligned and competing individual performance objectives are a root cause of organization dysfunction. If you need help fixing the conference room projector 45 minutes before your meeting, for example, it will not be useful for IT to process your support request tomorrow, even though their key performance indicator (KPI) is number of requests completed within 48 hours. The timeliness of your need, which affects your KPI, is not reflected in the KPIs of other members of your organization.

Here we propose a simple way to unify assessments across an organization when members of the organization are assessed along different dimensions. We leave aside the (significant) challenge of determining these assessment metrics. Indeed, many roles may be difficult to assess clearly and organizations can be misled by ill-informed KPIs. Instead, we focus on a way to ensure that different assessment metrics (KPIs), when applied differently across the organization, do not lead to diverging objectives or competing incentives.

“Secret Santa” assessment

Suppose $m$ key performance indicators (KPIs) has been developed for an organization of $N$ members. Let $\mathbf{k}_i \in \mathbb{R}^m$ be an $m \times 1$ vector representing the performance of member $i$ according to these KPIs. Of course, each member will be focused on only a subset of KPIs relevant to their organizational role, so we can think of $\mathbf{k}_i$ as being a sparse vector. Let $s_i : \mathbb{R}^m \to \mathbb{R}$ be a scoring function assigned to $i$. Suppose this scoring function is used by the organization to assess member performance; it may be tailored to the duties of $i$ as necessary, and can consider only the subset of KPIs relevant to $i$.

Ideally, different scoring functions should align with the overall goals of the organization so members are incentivized to collaborate. However, competing incentives may form when different members have different scoring functions that are not sufficiently aligned. For such situations, we introduce a simple process to more strongly couple or unify incentives across an organization without having to replace or modify the existing KPI system.

For each member $i$, pair $i$ with $n$ ($1 \leq n \leq N-1$) other members $j$ chosen uniformly at random. Then, define a new score $\sigma_i$ that acts as a scalarized combination of $i$’s own score and the average score of the randomly paired members:

$$\sigma_i = \alpha s_i (\mathbf{k}_{i}) + \left(1-\alpha\right) \frac{1}{n}\sum_{j} s_j(\mathbf{k}_j) \tag{1}$$

where $\alpha \in [0,1]$ and the sum runs over the $n$ members $j$ that were randomly paired with $i$.

Discussion

Why combine scoring functions in this manner, particularly using random pairing? The new score $\sigma_i$ aligns $i$ both with their KPIs and the KPIs of other members $j$, with $\alpha$ controlling the relative strength. By coupling member performance the hope is that all members will be more strongly incentivized to help one another, lowering competition between members and between divisions.

Random pairing, as opposed to the design of a pairing network specific to the organization, allows for this assessment strategy to be implemented without additional time or effort. In other words, Eq. 1 serves as an information-free coupling of member incentives. Random sampling also helps ensure fairness and prevent perverse incentives: all possible pairings are equally likely and members will not benefit much from “gaming the system” by optimizing their paired KPI scores. We also anticipate each member assessment, for example, quarterly assessments, will be computed using a fresh set of randomly paired $j$’s. Finally, to further avoid perverse incentives, members would not know who they are paired with nor the individual paired scores $s_j$, but only the average score.

Equation 1 has two parameters, $\alpha$ and $n$. We anticipate $\alpha \geq 1/2$ to be appropriate, placing most weight on the member themself. Meanwhile, $n$ controls how much of the organization impinges on an individual. When $n = N-1$ there is maximum coupling—essentially, a single paired score for all members. Perhaps this would act similar to an external assessor, like a stock price. On the other hand, if $n = 1$, each member will be connected to only one other member and may be overly affected when being assessed if, for example, that member performed badly. We therefore anticipate smaller values of $n > 1$ to be most helpful, perhaps in the range 5–10. This value of $n$ provides coupling across the organization without forming a single metric such as stock price (which would provide no per-member scoring information) while also averaging over fluctuations so that each member is not overly affected by a lucky or unlucky pairing.

Of course, there are likely serious flaws with this idea that we have not yet considered. Indeed, we are not currently certain whether this is even a novel idea.