Your early-phase fine-motor data came back variable. Here is why, and what to fix before the next study.
Variable fine-motor and tremor data in Phase 1 or Phase 2 almost always traces to the measurement, not the biology: rater variability, ordinal compression, practice effects, assessment timing, inconsistent instructions, and device differences. Each has a distinct signature in the data and a distinct fix. This guide walks through all six, then shows what the numbers look like when the endpoint is computed instead of rated.
By David McLaughlin, Founder, Cerebral Innovations. 40+ CNS clinical trials; co-author of the scoping review of 120 spiral analysis studies.
The six variance sources, and how to recognize each
Diagnose before you redesign. Each row gives the mechanism, the signature to look for in your existing data, and the change that removes it from the next protocol.
Rater variability
Different raters, and the same rater on different days, score the same performance differently. On a 0 to 4 item, a one-point disagreement is 25% of the scale.
Between-rater ICC below 0.9; site-level mean differences; scores that drift over the study.
Central rater training and calibration for the anchor scale, plus a computed endpoint that has no rater in the loop.
Ordinal compression
Ordinal items have floors and ceilings and cannot register a 15% change in amplitude. Composite scores dilute a fine-motor effect with items the compound does not affect.
Change scores clustering at 0 and 1; responders defined by a single item shift.
A continuous metric matched to the mechanism (frequency, amplitude, velocity, pressure) pre-specified as the exploratory endpoint.
Practice effects
Patients get better at a task through repetition. Without control, early-visit improvement is indistinguishable from drug effect and inflates placebo-arm change.
Improvement in the placebo arm on task-based measures; largest gains between visits 1 and 2.
Familiarization trials, a run-in baseline to plateau, and metrics chosen for robustness to learning. See the protocol below.
Timing relative to dosing
Motor state in Parkinson disease swings with medication. An assessment 40 minutes before or after a dose is a different measurement.
Variance that tracks visit time of day or hours since last dose.
Fixed assessment windows relative to dosing written into the protocol and captured in the data.
Instructions
Telling a patient to draw quickly versus accurately changes velocity and pressure metrics. 63% of the 120 studies in the scoping review did not state which instruction was given.
Site-level differences in speed metrics; inconsistent task durations.
A scripted instruction, identical at every site and visit.
Devices and environment
Different tablets, sampling rates, styluses, table heights, and seating change the raw signal.
Site-level offsets in geometry or pressure metrics; step changes when hardware is swapped.
Locked device configuration per site with quality flags for cross-device interpretability (NeuroQuantix samples at 240 Hz on iPad Pro and Wacom digitizer).
What the numbers look like when the endpoint is computed
A rated endpoint carries rater error and ordinal compression in every observation. A computed kinematic endpoint carries neither. The published contrast for essential tremor:
- ICCTETRAS inter-rater reliability of ICC 0.74 to 0.85 (Elble & Ellenbogen, 2017, Tremor and Other Hyperkinetic Movements) versus test-retest reliability up to icc 0.97 for the velocity spectral peak metric in essential tremor (Haubenberger et al., 2011, Movement Disorders).
- NAs few as 9 subjects per arm at 95% power to detect a 50% tremor reduction with the velocity spectral peak metric, versus roughly 25 to 33 per arm with TETRAS (Haubenberger et al., 2011).
These are literature values for specific metrics and populations, not guarantees for your compound. The point is the direction and the size of the gap: reliability above 0.95 changes the enrollment math by a factor of three or more.
Run the math for your effect sizeControlling practice effects, specifically
A standardized administration protocol that controls practice effects and rater dependence and preserves sensitivity to treatment effects.
- Fixed familiarization trials before the scored trialLearning effects on repeated spiral drawing were flagged in published sponsor pilots; scoring only after a fixed number of practice spirals moves the measurement past the steepest part of the learning curve.
- Standardized speed-versus-accuracy instruction63% of the 120 reviewed studies did not state whether patients were told to prioritize speed or accuracy. The instruction changes velocity and pressure metrics, so it is fixed and scripted.
- Run-in baseline to a stable score before randomizationA short baseline series establishes each patient's plateau so that change from baseline reflects treatment rather than task learning.
- Pre-specified metrics chosen for robustness to learningTremor spectral metrics (frequency, amplitude) are less sensitive to task learning than time or geometry metrics; the endpoint is pre-specified per indication and mechanism.
Written into the protocol and the statistical analysis plan before first patient in.
What to change before the next study
- 1Keep the anchor scale
TETRAS, MDS-UPDRS, or SARA stays. It carries the regulatory precedent and clinical interpretability. Calibrate its raters centrally.
- 2Add a computed exploratory endpoint
Pre-specify one kinematic metric matched to the mechanism, assessed at the same visits. Now you have a within-program comparison of sensitivity.
- 3Write the protocol controls
Familiarization trials, scripted instruction, dosing-relative timing windows, locked device configuration, run-in baseline.
- 4Pre-register the analysis
Change from plateau baseline, not from visit 1. Domain-specific metrics reported separately, not collapsed into a composite.
Frequently asked
Why is fine-motor data so variable in early-phase CNS trials?
Six sources, usually stacked: inter-rater and intra-rater variability on ordinal scales, ordinal compression that cannot register small change, practice effects on repeated tasks, assessment timing relative to medication state, unstandardized task instructions, and device or environment differences between sites. Small early-phase samples make each of these visible as noise rather than averaging it out.
How can I tell whether the noise is the rater or the patient?
Partition the variance. Compare within-rater retest agreement to between-rater agreement on the same videos or drawings, then compare both to the between-visit change. If between-rater variance approaches the treatment-effect size you are looking for, the instrument, not the patient, is limiting the study. TETRAS inter-rater reliability is ICC 0.74 to 0.85 in published work, which leaves substantial error variance.
Does adding a digital endpoint fix this in Phase 2?
It addresses the rater and compression components directly, because kinematics are computed rather than rated and are continuous rather than ordinal. Timing and instruction problems are handled by the standardized administration protocol. Run the objective measure beside the rating scale as a pre-specified exploratory endpoint so you can quantify the difference in your own population.
Should we change the primary endpoint after a noisy Phase 1?
Rarely. Keep the regulatory anchor and add precision beside it. A noisy Phase 1 is an argument for pre-specifying a more sensitive exploratory endpoint in Phase 2, tightening the protocol around timing and instructions, and building the within-program evidence that would justify promoting the objective measure later.
What is the fastest way to get an outside read on our early-phase motor data?
Send a de-identified variance summary (rater agreement, retest agreement, and change scores by visit) and we will tell you which variance source dominates and what to change. This review is free and takes one call.
Related: questions to ask before adopting a digital motor endpoint and how the digital motor endpoint options compare.