1/5 One example of modeling changes in "a single item" over time is given by reinforcement learning. This series of blogs details a simple example: http://haines-lab.com/post/2017-04-04-choice_rl_1/ https://twitter.com/aaronjfisher/status/1281663024655241216

2/5 The basic idea being that we can model learning dynamics using observed preferences among two choices, each with different average payoffs (1 choice is better on average, but this must be learned). We can simulate behavior with these models: http://haines-lab.com/post/2018-03-24-human-choice-and-reinforcement-learning-3/

3/5 And then we can try estimating the parameters from the simulated data to determine if we can accurately, and precisely, recover the "true" parameters. This is akin to assessing the reliability of our measure, but before we have even collected empirical data.

4/5 The take-away being that we can build models of the temporal dynamics of whatever behavior we are interested in, and we can use simulations to determine if we can reliably estimate parameters from these person-specific models given the number of observations we have, etc.

5/5 From this perspective, increasing the number of different items to ensure high reliability is just a heuristic. What really matters is that we can estimate data-generating parameters (or "true scores", see http://haines-lab.com/post/2020-06-13-on-curbing-your-measurement-error/) with enough precision for a given inference.