Robotic Rigor: Validity of the Kinarm End-Point Robot Visually Guided Reaching Test in Multiple Sclerosis.

Multiple sclerosis (MS) is an inflammatory and demyelinating disease of the central nervous system. 1 Pervasive pathophysiology results in heterogeneous symptomology, including sensorimotor, cognitive, and emotional impairments that negatively impact quality of life and increase demand for medical resources. 2 , 3 No less than 19 medications alleviate MS-related relapses, but none ultimately cure the hallmark of the disease, demyelination. 4 MS progression is often covert, beginning up to 10 years before diagnosis 5 , 6 and occurring in the absence of relapses. 7 Novel measures and technologies 8 that track subtle MS-related performance changes have 9 , 10 , 11 and will 12 enhance disease understanding while continuing to test intervention strategies aiming to halt disease progression and restore function.

Robotic research has increased exponentially in recent years; the PubMed MeSH term “robotics” indicates that the yearly articles published from 2012-2022 have more than doubled. Interactive robotic tools are designed to assess motor, sensory, and cognitive function within a virtual environment. 13 Previous robotics research has aimed to understand upper-limb function in healthy and clinical populations, including stroke, 14 epilepsy, 15 and more. 16 Certain robotic tools derive metrics from “standard tests,” including a consolidated “task” or overall score and precise spatiotemporal values to help delineate specific movement characteristics.

Visually guided reaching (VGR) is a motor-centric robotic test; it may be particularly sensitive for characterizing the heterogeneity of MS-related upper-limb motor impairment. Two previous studies provide conflicting evidence regarding the VGR test as a valid upper-limb impairment measurement in MS. Simmatis et al 17 compared 8 standard robotic tests to Nine-Hole Peg Test (9HPT) 18 scores in 43 MS persons with mild hand problems. 19 The authors demonstrated face validity for overall task scores, but the relationship strength varied considerably depending on the robotic test and clinical variable. 17 For example, 9HPT held a moderate correlation with the object hit and avoid test but a weak correlation with the VGR test. 17 Unfortunately, the authors did not report on specific spatiotemporal variables for each of the 8 robotic tests. 17 Wijeyaratnam et al 20 considered spatiotemporal values during a VGR test in 24 persons with MS. Except for 1 spatiotemporal value (ie, angular error variability), the authors found no significant differences between the group (half [n=12]) that reported upper-limb impairment versus the group that did not. Although the authors 20 collected 9HPT, they did not analyze it relative to VGR outcomes. Moving forward, it will be essential to determine whether a robotic measurement tool like the VGR test detects performance differences, even when impairments are mild and potentially difficult to ascertain using conventional tests, such as 9HPT; gait research indicates that spatiotemporal values beyond completion time (eg, walking speed)—the primary outcome for the 9HPT—offers further insight for identifying and understanding individuals with mild impairments. 21 , 22 , 23

To this end, we analyzed: (1) convergent (9HPT); and (2) criterion (status based on subjective rating of hand problems) validity of a robotics VGR test by considering the task score and movement planning and correction spatiotemporal variables. We hypothesized that the: (1) VGR task score and spatiotemporal variables of movement planning and correction would demonstrate moderate to strong correlations with 9HPT (good convergent validity); and (2) the VGR test would differentiate between MS persons who do and do not self-report hand problems (good criterion validity).

Methods

Design and participants

We conducted a cross-sectional validation analysis of data collected from consecutive individuals attending a specialized MS clinic. 24 To be included in the study, we required participants to complete the: (1) a Kinarm End-Point VGR Test; (2) 9HPT; and (3) question 15 (ie, hand problems question) from the Multiple Sclerosis Impact Scale (MSIS-29; more details below). 25 Potential participants also satisfied the following criteria: (a) diagnosed with MS according to the 2010 26 or 2017 27 McDonald criteria; (b) older than 18 years; and (c) stable disease being relapse-free during the previous 3 months. We collected demographic data through a combination of health records and in-person assessments. The institutional ethics board approved the study (HREB#2015.103), and we obtained informed written consent according to the Declaration of Helsinki. We aimed to achieve a sample size of 40, comparable to previous studies testing Kinarm VGR in MS. 17 , 20

Assessments

Kinarm End-Point VGR Test

Before beginning each session, we calibrated the Kinarm End-Point Lab and Dexterit-E 3.8.2 software ( fig 1 A) according to the manufacturer's instructions. In brief, the VGR (4-target) test requires participants to move their hand as quickly and accurately as possible from a central target to 1 of 4 peripheral targets ( fig 1 B); peripheral targets are 10 cm from the central target, spaced 90° apart, and presented in pseudo-random order; a single trial involved moving from the central to the peripheral target. 28 After reviewing the instructions and ensuring comprehension, participants performed 24 trials per hand, including 5 trials per peripheral target, plus 4 “catch” trials in which no peripheral target was presented. Completion of the VGR test took ∼10 minutes. Hand dominance was self-reported, and we randomized the between-participant testing order (ie, dominant vs nondominant) to prevent a practice effect. 29 We analyzed the task score, an overall or summary score based on all available VGR performance metrics 30 , 31 ; the task score is a z-score normalized to “healthy” adults of the same age and sex. We also considered 2 spatiotemporal domains: (1) movement planning via reaction time and initial direction angle; and (2) movement correction via movement time and path length ratio ( table 1 , fig 2 ). We reported spatiotemporal values in their raw format (ie, degrees and/or cm).

Kinarm End-Point VGR Test. (A) Kinarm End-Point Lab. (B) Screenshot of the VGR test with all peripheral targets illuminated. Note that only 1 peripheral target illuminates during an actual trial.

Table 1 Objective description of spatiotemporal values for the Kinarm End-Point VGR Test Table 1 Category Value Reporting Definition Explanation Movement planning Reaction time Median value of all trials. Time between illumination of the end (ie, peripheral) target and movement onset. Response speed and efficiency in movement initiation. Lower value indicates a better score. Initial direction angle Median value of all trials. Angle between line 1 and line 2, where: Line 1 = straight line distance from the central target to the hand position after the “initial movement phase.” Line 2 = straight line distance from central to peripheral target (ie, shortest possible path) Precision and coordination in movement initiation. Lower value indicates a better score. Movement correction Movement time Median value of all trials. Total time elapsed from movement onset to movement offset or, generally, when the end (ie, peripheral) target is achieved. Speed and efficiency in task completion, beyond movement initiation. Lower value indicates a better score. Path length ratio Mean value of all trials. Ratio of line 1 to line 2, where: Line 1 = total distance traveled from movement onset to movement offset. Line 2 = straight line distance from the central to peripheral target (ie, shortest possible path) Precision, coordination, and trajectory, beyond movement initiation. Values are 1.00+. Values closer to 1.00 indicates a better score. NOTE. Definitions are adapted from the manufacturer's documentation. Figure 2 provides a graphical description of each spatiotemporal value.

Objective description of spatiotemporal values for the Kinarm End-Point VGR Test

NOTE. Definitions are adapted from the manufacturer's documentation. Figure 2 provides a graphical description of each spatiotemporal value.

Graphical description of spatiotemporal values for the Kinarm End-Point VGR Test. (A) Illustrates initial direction angle and path length ratio, where: green (d) = actual hand path while reaching from the central (white) to the peripheral target (red); white (l) = shortest possible hand path while reaching from the central to the peripheral target; a° = initial direction angle; and d/l = path length ratio. (B) Illustrates the reaction and movement time phases relative to hand speed and test completion time. Table 1 provides an objective description of each spatiotemporal value.

Nine-Hole Peg Test

The 9HPT is considered a criterion standard measure of upper-limb impairment in MS. 18 In brief, participants remove pegs from a container, one by one, and place them into holes on a board. Then, return the pegs to the original container. Participants performed 2 attempts on each hand, and the final score via completion time(s) is averaged for the dominant and nondominant sides. Like the Kinarm, we randomized the testing order (ie, dominant vs nondominant) to prevent a practice effect.

Subjective rating of hand problems

The MSIS-29 is a 29-item questionnaire examining the impact of MS on physical and psychological functioning. The MSIS-29 is a reliable, sensitive, 32 and commonly cited tool. 25 We specifically used question 15 from the MSIS-29: “In the past 2 weeks, how much have you been bothered by difficulties using your hands in everyday tasks”? We dichotomized our sample into study participants who reported “no hand problems” (Q15=1 [Not at all]) and “some hand problems” (Q15=2 [A little]-5 [Extremely]).

Statistical analyses

Except for sex and hand dominance, we summarized participant descriptors using means and standard deviations and outcome data as the median and interquartile range. We identified outliers as those ±3 times the interquartile range. Visual inspection of histograms and a Shapiro–Wilk normality test confirmed that our outcome data were not normally distributed. As such, we leveraged nonparametric tests. Specifically, we used a Spearman rank-order correlation to measure the relationships between Kinarm End-Point VGR variables and 9HPT (outcome number 1 - convergent validity). Additionally, we used a Mann–Whitney U test to examine differences between those who self-reported “some” versus “no” hand problems (outcome number 2 - criterion validity). Using a chi-Square and Mann–Whitney U test as appropriate, we examined group (ie, “some” vs “no” hand problems) characteristics. We executed all statistical tests using SPSS ( b version 29; IBM Canada Ltd).

Results

Demographics and outliers

Fifty percent (n=21) of the study participants reported having “some hand problems.” Groups were similar in their characteristics (Some: mean age, 52.52±10.69 y; females [n=16]; disease duration, 18.81±10.38 y vs None: mean age, 51.24 ± 12.73 y; females [n=14]; disease duration, 17.71±10.16 y). The only significant between-group difference was that the group reporting “some hand problems” were more often right-handed ( P =.038; table 2 ). Both groups scored in the “abnormal” range, given that their average 9HPT completion times were between 18.00 and 32.99 seconds. 19

Table 2 Group characteristics Table 2 Characteristic None (n=21) Some (n=21) P Value Age (y) 51.24±12.73 52.52±10.69 .706 Females ( n ) 14 16 .495 Height (m) 1.66±0.07 1.62±0.17 * .573 Weight (kg) 83.55±43.82 81.98±27.96 * .748 Body mass index 29.69±13.84 31.26±9.60 * .333 Right-hand dominance ( n ) 15 20 .038 Disease duration (y) 17.71±10.16 18.81±10.38 .811 Dominant hand  9HPT (s) 21.50 (19.60-24.29) 23.70 (20.27-28.39) .110  VGR task score 0.9790 (0.5830-2.0315) 1.4500 (1.1045-2.9315) .105  VGR reaction time (s) 0.2810 (0.2605-0.3215) 0.3400 (0.2735-0.3725) .036  VGR initial direction angle (°) 0.0240 (0.0210-0.0430) 0.0300 (0.0220-0.0505) .371  VGR movement time (s) 1.1150 (1.0175-1.4405) 1.1830 (1.0415-1.2595) .980 VGR path length ratio (cm/cm) 1.0700 (1.0530-1.0895) 1.0770 (1.0575-1.1090) .339 Nondominant hand  9HPT (s) 23.00 (20.47-25.18) 23.52 (20.33-28.98) .458  VGR task score 1.1830 (0.8055-2.6670) 1.4900 (0.9035-2.5835) .660  VGR reaction time (s) 0.2940 (0.2710-0.3295) 0.3060 (0.2775-0.3570) .358  VGR initial direction angle (°) 0.0340 (0.0260-0.0450) 0.0310 (0.0255-0.0445) .696  VGR movement time (s) 1.1200 (0.9900-1.2845) 1.0780 (1.0075-1.2165) .580  VGR path length ratio (cm/cm) 1.0840 (1.0655-1.1275) 1.1070 (1.0590-1.1425) .414 NOTE. Except for sex and hand dominance, participant descriptors (ie, age to disease duration) are reported as mean ± standard deviation. Conversely, outcomes (ie, 9-hole peg test and visually guided reaching) are reported as the median (interquartile range). Using a chi-Square and Mann–Whitney U test as appropriate, we examined group (ie, “some” vs “no” hand problems) characteristics. Bolded and italicized P values indicate statistical significance. ⁎ n=19.

Group characteristics

NOTE. Except for sex and hand dominance, participant descriptors (ie, age to disease duration) are reported as mean ± standard deviation. Conversely, outcomes (ie, 9-hole peg test and visually guided reaching) are reported as the median (interquartile range). Using a chi-Square and Mann–Whitney U test as appropriate, we examined group (ie, “some” vs “no” hand problems) characteristics. Bolded and italicized P values indicate statistical significance.

n=19.

We identified 2 9HPT outliers, 1 for both hands and another for the nondominant side. The full sample (n=42) is presented within, whereas the results minus the 2 outliers (n=40) are presented as supplemental material (available as an Appendix with Tables online only at http://www.archives-pmr.org/ ). Values (ie, correlation coefficient, P values, etc) change slightly, but the removal/inclusion of outliers had no meaningful impact on the results.

Relationships between Kinarm VGR and 9HPT

Dominant-side 9HPT demonstrated a moderate correlation with VGR task score ( r =0.500, P <.001) and with the movement planning variables reaction time ( r =0.489, P =.001) and initial direction angle ( r =0.429, P =.005; fig 3 A). Similarly, on the nondominant side, 9HPT demonstrated a moderate correlation with VGR task score ( r =0.410, P =.007), reaction time ( r =0.521, P <.001) and initial direction angle ( r =0.321, P =.038; fig 3 B). No Kinarm VGR movement correction values (ie, movement time or path length ratio) significantly correlated with 9HPT on either side.

Kinarm End-Point VGR task score and movement planning spatiotemporal values (ie, reaction time and initial direction angle) are significantly associated with 9HPT time on the dominant (A) and nondominant (B) side.

Kinarm VGR and 9HPT relative to self-reported hand problems

VGR reaction time for the dominant hand significantly differed between the 2 groups, with the “no hand problems” group being faster than the “some hand problems” (0.2810 [0.2605-0.3215] vs 0.3400 [0.2735-0.3725] s; P =.036; table 2 , fig 4 ). No other measures significantly differed between the 2 groups, including all other VGR outcomes and the 9HPT.

Kinarm End-Point VGR mean reaction time of the dominant but not the nondominant hand is significantly different between the 2 groups (ie, none vs some hand problems).

Discussion

We explored the relationships between established objective clinical measures (ie, 9HPT), self-reported hand problems (ie, question 15 MSIS-29), and novel robotic-obtained values (ie, Kinarm End-Point VGR Test) in our study participants. We report 2 key findings, both of which partially support our hypotheses. First, task score and movement planning (ie, reaction time and initial direction angle), but not movement correction (ie, movement time and path length ratio), spatiotemporal values exhibited moderate correlations with 9HPT. Thus, the VGR task score and movement planning values showed modest convergent validity, with the most compelling results in the movement planning variable reaction time. Second, VGR reaction time (in the dominant hand), but not the 9HPT, differentiated between persons self-reporting “some” versus “no” hand problems. As such, VGR reaction time achieved good criterion validity. Our findings suggest a meaningful relationship between 9HPT and the Kinarm End-Point VGR movement planning values, but the latter, specifically reaction time, is more sensitive in distinguishing self-reported hand problems.

Like Wijeyaratnam et al, 20 we found that a dominant hand VGR movement planning spatiotemporal value differentiated between persons with MS who did and did not self-report upper-limb impairment. Therefore, our work and Wijeyaratnam et al 20 highlight the importance of movement planning over movement correction in MS. Importantly, we found the specific movement planning spatiotemporal value, reaction time, differentiated between those that did and did not self-report, whereas Wijeyaratnam et al 20 found the movement planning spatiotemporal value, absolute change in angular error variability (°), to do so. We are unsure why our findings differ from Wijeyaratnam et al, 20 but we can speculate. Wijeyaratnam et al 20 likely included individuals with worse disability, given that their average 9HPT times were ∼4-5 seconds slower than our sample. Another critical between-study difference is the question asked regarding subjective upper-limb dysfunction. Such granular between-study similarities/differences highlight the need to understand the subtle clinical and physiological transitions in MS progression.

Other groups have used alternative virtual reality and/or robotic techniques to explore spatiotemporal properties in MS-related upper-limb impairment and, in doing so, attain further insight than the single outcome (ie, completion time) offered by the 9HPT. 33 , 34 , 35 In alignment with our findings, the collective work 33 , 34 , 35 suggests that robotic/virtual reality instruments correlate with 9HPT performance but are capable of capturing data invisible to the 9HPT and/or are more tightly aligned with other clinical outcomes. Unfortunately, such previous work 33 , 34 , 35 did not measure reaction time, but Lambercy et al 33 and Corona et al 34 both highlighted “less smooth” movement in their study participants, which may be considered analogous to our movement correction outcomes, particularly path length ratio, where we found no significant results. Similar to Wijeyaratnam et al, 20 drawing a direct comparison between our findings and Lambercy et al 33 and Corona et al 34 is difficult given that their participants were likely experiencing more severe disease progression as per higher (ie, worse) 9HPT time. In weighing the findings of the Kinarm work (ie, our study and Wijeyaratnam et al 20 ) against other virtual reality/robotic instruments (ie, Lambercy et al 33 and Corona et al 34 ), future research exploring spatiotemporal properties in the Kinarm should compare “movement planning” against “movement correction” along the MS classification spectrum, inclusive of a healthy comparator. Within movement planning, reaction time should be afforded special attention given that in addition to our findings, it was recently implicated in driving performance for “early stage” MS. 36 Furthermore, researchers regularly assess reaction time in executive function, 37 , 38 , 39 a cognitive domain that makes us uniquely human 40 , 41 because of its role in higher-order (ie, goal-setting, planning, etc) processing. 42 , 43 , 44

Using the Kinarm VGR test, we propose that the movement planning variables are driving the task score-9HPT relationship, given that the task score is a cumulation of the spatiotemporal values and independently, movement planning (but not movement correction) spatiotemporal values demonstrated a moderate correlation with 9HPT. Our results support that rehabilitation targeting movement planning, particularly reaction time, may help improve overall upper-limb function, considering that 9HPT performance translates to one's ability to complete activities of daily living with the upper limb (ie, reach, grasp, etc). 45 However, the 9HPT, because it is timed with a stopwatch, is unable to discriminate between specific movement impairments when compared to kinematics or robotics. 33 , 34 , 35 Arguably, robotic-derived spatiotemporal values would help to better define the specific movement rehabilitation target (for instance, tremor or delayed reaction time) compared to an overall score provided in seconds.

It is unclear why Kinarm End-Point VGR reaction time was the only objective measure to differentiate between our 2 groups, but we can speculate. Relative to other Kinarm spatiotemporal values, reaction time demonstrated the lowest between-participant variability or error, as reflected in the standard deviation values. For context, the standard deviation for reaction time ranged from 14%-18% of the mean. In contrast, the standard deviation for the initial direction angle was > 100% of the mean in some instances. The robotic nature of the Kinarm makes it possible to accurately capture such variability while precisely measuring spatiotemporal values at the millisecond level; the significant finding for the Kinarm VGR reaction time was due to a between-group difference of 350 milliseconds. Perhaps 9HPT reaction time also significantly differs between groups at the millisecond level, but humans cannot independently acquire such precise values; this precision further emphasizes why robotic measurements have grown exponentially in recent years. Such growth is supported by other fields, such as gait or walking, where researchers have consistently demonstrated that spatiotemporal values collected via electronic walkway mats support 21 , 22 and/or provide further insight 23 than just gait speed. Additionally, spatiotemporal values from bipedal hopping on such electronic mats can detect covert sensorimotor dysfunction in study participants showing “normal” neurologic examinations. 10 , 11

We found evidence of laterality or hand dominance given that our groups (ie, some versus no hand problems) significantly differed in: (1) hand dominance; and (2) Kinarm End-Point VGR reaction time of the dominant but not the nondominant hand. Why humans have a dominant side or laterality and how it develops is not entirely understood, 46 but it typically reflects the stronger, faster, and more dextrous side. 47 , 48 When asking about hand problems, we followed question 15 from the MSIS-29. However, the question does not specify the affected side nor how it relates to dominance. Such details may have clarified a dominant-affected side interaction, providing further insight into the subjective experience of hand problems or why our participants were assigned to a specific group. At the very least, such results underscore MS disease heterogeneity and the need to understand the underlying physiology and clinical manifestation of (upper-limb) dysfunction in the dominant and nondominant side.

Overall, our findings suggest the Kinarm is a clinician-ready tool useful for detecting MS-related upper-limb dysfunction; this may be particularly true for the prodromal stage, characterized by an increase in the subjective reporting of new health issues that are difficult to ascertain. 5 , 7 We provide such a recommendation while acknowledging that the 9HPT is more accessible than the Kinarm. At the very least, the Kinarm End-Point VGR Test will augment 9HPT and the participant's/patient's subjective experience to create a more complete picture of MS-related upper-limb dysfunction. Looking into the future, the Kinarm (and other robotic/electronic platforms) may clarify disease progression, identify those at greater risk of future decline before clinical manifestation, the degree to which such risks are sex-specific, and elucidate intervention strategies to delay disease progression.

Study limitations

We only demonstrated modest convergent validity in most of our results, with the strongest being reaction time, which also demonstrated good criterion validity. However, validity thresholds are a complex topic with divergent opinions. 49 If our correlations were strong to very strong (ie, >0.6), we might have increased our convergent validity at the expense of criterion validity, therefore losing our second main finding. Instead, the moderate correlations suggest these tests assess overlapping but distinct dimensions of motor performance, such that Kinarm properties offer a more granular view of motor function (eg, precision, timing), while the Nine-Hole Peg Test captures overall dexterity. Ultimately, the Kinarm can likely capture subtle motor impairments that are not as easily detected by the traditional criterion standard. This may be particularly meaningful given that our sample focused on individuals with covert MS, where disease markers and progression are subtle and not entirely understood. 5 , 7 As such, the moderate correlations may have been the best we could expect between Kinarm and 9HPT.

It is also important to highlight that all our participants were diagnosed with MS, but unlike a younger MS individual, an older individual is more likely to be suffering from additional (age-related) comorbidities that interact with MS and/or affect the outcomes we collected. None of our participants had significant upper extremity arthritic or movement limitations, but reaction time does slow with age 38 ; how age-related diseases and syndromes interact with 50 , 51 and without 52 MS is an ongoing area of research. Despite the challenges of recruiting a clinical population, 53 , 54 future work may wish to confirm our findings in a larger sample size. In addition to those in the early stages of MS, it would be beneficial to include the MS disease spectrum, inclusive of “healthy” controls and those with more severe disease progression. In time, the Kinarm could progress so that a diverse spectrum of normative data are available for age and sex, as is the case for the 9HPT. 55 Further, it will be essential for future studies to leverage a larger sample when aiming to understand Kinarm validity relative to sex differences. 56 Our sample precluded us from executing a sex-specific analysis, but a rapidly growing mass of literature indicates sex differences exist at the molecular, 57 structural, 52 and behavioral levels. 58 Finally, criterion validity is usually reserved for a criterion standard measure, but we used a subjective report; we took this approach to identify tools that may be more sensitive and, therefore, insightful than 9HPT.

Conclusions

Kinarm VGR movement planning spatiotemporal values demonstrated modest convergent validity given the moderate correlations with 9HPT. Additionally, the mean reaction time of the Kinarm End-Point VGR Test demonstrates good criterion validity as it differentiated between-study participants self-reporting “some” versus “no” hand problems. Our results suggest that the Kinarm is a valid measure of upper-limb dysfunction in our study participants, and it may be more insightful than the objective criterion standard 9HPT. Furthermore, focusing on just task score may be limiting and movement planning spatiotemporal values appear more meaningful than movement correction and could, therefore, be a target for future intervention strategies.

Authorship contribution

NWB: conception and design, data analysis and interpretation, writing—original draft, final approval of published version. SR: conception and design, data acquisition, data analysis and interpretation, writing—review and editing, final approval of published version. JRA: data analysis and interpretation, writing—review and editing, final approval of published version. CJN: data acquisition, writing—review and editing, final approval of published version. MP: conception and design, data analysis and interpretation, writing—review and editing, final approval of published version.

Suppliers

a Kinarm.

b SPSS version 29; IBM Canada Ltd.

Disclosures

The authors declare no competing interests.

Footnotes

The research was supported by The Canadian Institutes of Health Research, Grant Numbers 169649, 173526 (MP), Newfoundland and Labrador Research and Development Corporation, Grant Number 5404.1699.104 (MP), Canada Foundation for Innovation Grant Number 33621 (MP), Canada Research Chairs Program (950-232532; MP). NWB is supported by a Canadian Institutes of Health Research Fellowship (FRN: 489847).

Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.arrct.2024.100382 .

Contributor Information

Nick W. Bray, Email: nwbray@mun.ca.

Michelle Ploughman, Email: Michelle.Ploughman@med.mun.ca.

Supplementary materials

References

Associated Data

Supplementary Materials