CoGames: Разработка адаптивного инструмента мониторинга на основе смартфонов и gamified для когнитивной функции в мультипликационном склерозе.

Introduction

In multiple sclerosis (MS), a chronic inflammatory and neurodegenerative autoimmune disease affecting the central nervous system, up to 70% of patients display signs of cognitive impairment (CI), often negatively influencing emotional well-being, quality of life, and working capacity [ 1 – 5 ]. Still, comprehensive monitoring of cognitive functions is not a standard procedure in most clinics [ 6 ]. This can mainly be attributed to the fact that such assessments are time-consuming, costly, and often burdensome for patients [ 6 ]. Novel assessment tools that are well accepted, convenient, and sensitive to change are needed. Digital devices such as computers, tablets, and smartphones have been used as a basis for medical rehabilitation and assessments for several years [ 7 – 16 ]. Especially smartphones with their wide distribution in the general population, high-quality sensors, and user-friendliness have great potential to contribute to accurate, remote, and unbiased assessment [ 17 ].

In parallel to the emerging use of digital devices in the medical field, the inclusion of gamification elements in medical devices has gained popularity as well [ 9 , 10 , 15 , 18 ]. Using motivational gamification elements in an assessment tool might improve adherence and motivate patients to reach their maximum performance [ 19 , 20 ]. For example, a scoring and reward system can elicit feelings of competition and accomplishment [ 20 , 21 ]. Furthermore, a dynamic difficulty adjustment (DDA) system comprising multiple difficulty levels can prevent frustration and boredom and promote a flow state, a state of high concentration [ 22 , 23 ].

We hypothesize that well-designed smartphone games might be the optimal tool to circumvent the shortcomings of current established neuropsychological assessments [ 18 , 24 ]. To further investigate this and as part of the development of a smartphone-based app for monitoring MS disease activity and progression (dreaMS, NCT05009160 , BASEC-ID: 2021- D0040), we developed a set of multi-leveled smartphone games to assess cognitive function. As a first step, the CoGames study aimed to investigate the relative difficulty between levels, reliability across difficulty levels, estimate potential practice effects on performance over time, and assess their acceptance.

Methods

In cooperation with the medical device software manufacturer Indivi Ltd. (a DBA of Healios AG), we developed a set of smartphone games for the assessment of cognitive function [ 25 , 26 ]. The six games cover five cognitive domains: working memory, information processing speed, short-term memory, psychomotor speed, and mental flexibility (set-shifting between two cognitive tasks). Information processing speed, as the most frequently affected cognitive domain in PwMS, is assessed in two games [ 27 ].

Standard protocol approvals, registrations, and patient consents

The CoGames study was conducted according to the standards of the World Medical Association Declaration of Helsinki and approved by the local ethics committee: Ethikkommission Nordwest- und Zentralschweiz (EKNZ), Basel, Switzerland (Req-2022–01571).

Study procedures

From April to July 2023, we recruited healthy volunteers via flyers and online advertisements in Basel, Switzerland. The participants were screened and instructed in a phone call. Information regarding the download of the app and the daily schedule was additionally sent by email. Participants were instructed to play the cognitive games autonomously at home on a daily basis for 11 consecutive days. The 11 days include performing all eight difficulty levels with one repetition per day for test–retest analysis and 3 days for completing the easiest level to assess change in performance over time. We decided to use a twice-daily frequency to keep the study duration as short and convenient to the participants as possible. To measure reliability via test–retest correlation, the participants were asked to play the same difficulty level of every game twice per day. Every day a different level was assessed, starting with the easiest and increasing difficulty daily, except for days 2, 6, and 11 where level 1 (“Beginner”) was repeated, to observe potential practice effects. To investigate the relative differences in difficulty between the levels, we applied linear mixed models with user as the random effect and difficulty levels, age, operating system, years of education, and their interaction with difficulty levels as fixed effects. The easiest level “Beginner” was used as the reference (intercept). Furthermore, we established a standardization allowing to compare scores across different difficulty levels. On days 4, 8, and 11 participants were asked to answer a short online questionnaire to assess overall acceptance and the impact of the different difficulty levels on acceptance. An overview of the study schedule is shown in Table  1 . After the completion of each game, the data were uploaded and stored on a general data protection regulation (GDPR)-compliant secure cloud (Google Firebase: https://firebase.google.com/docs/storage/ ). From there, the raw data were preprocessed and sent to the intranet servers of the University Hospital Basel. Table 1 Study schedule Task/day 0 1 2 3 4 5 6 7 8 9 10 11 Screening X Instructions X Questionnaire X X X Lvl.1 Beginner X X X X Lvl.2 Apprentice X Lvl.3 Graduate X Lvl.4 Expert X Lvl.5 Star X Lvl.6 Superstar X * * Lvl.7 Hero X Lvl.8 Deity X * Catch-a-Cloud only consists of six difficulty levels, and thus level 6 was repeated until the end of the study

Table 1 Study schedule Task/day 0 1 2 3 4 5 6 7 8 9 10 11 Screening X Instructions X Questionnaire X X X Lvl.1 Beginner X X X X Lvl.2 Apprentice X Lvl.3 Graduate X Lvl.4 Expert X Lvl.5 Star X Lvl.6 Superstar X * * Lvl.7 Hero X Lvl.8 Deity X * Catch-a-Cloud only consists of six difficulty levels, and thus level 6 was repeated until the end of the study

Study schedule

* Catch-a-Cloud only consists of six difficulty levels, and thus level 6 was repeated until the end of the study

Participants

Inclusion criteria were informed consent, age 18–80 years, owning and proficiency to use a smartphone, which met the technical requirements of the CoGames app (iOS ≥ 13.0, android ≥ 7.0). Exclusion criteria comprised any current or past neurological or psychiatric disease and any current dexterity or uncorrected vision impairments. Furthermore, participants using any medication or recreational drugs that are known to impact cognitive function or hand–eye coordination were not included.

Instruments and measurements

All cognitive games comprised eight difficulty levels except for the psychomotor game Catch-a-Cloud, which comprised only six levels. Measures that represent game performance were selected based on established neuropsychological tests: number of correct responses and percentage of correct connections. To keep the games as convenient and enjoyable as possible, we limited the game time to maximum 1 min except for the memory game Treasure Hunt and the psychomotor game Catch-a-Cloud. In Treasure Hunt we allow the participants to memorize and recall for as long as they need. Catch-a-Cloud lasts 50 s and is divided into two phases: 20 s Test Mode followed by 30 s Game Mode. The Test Mode contains fewer gamification elements, while in the Game Mode there are additional gamification elements, changing from level to level. All measures used for the analyses, brief game descriptions, and the cognitive domain they aim to assess are described in Table  2 . Screenshots of all games are shown in Fig.  1 : screenshots of the challenges included in CoGames. Example videos of all CoGames are included in Online Resource S2-7. Table 2 Domains, measures, and description of CoGames Cognitive domain Game (no. of levels) Primary measure Description Working memory Match maker 8 levels Number of correct responses in 60 s The system shows an either colorful or gray shape on the screen. With every answer, a new shape appears and the previous is hidden. The task is to continuously decide (“Yes”/”No”) whether the presented shape matches the one shown “n” taps before. “N” increases with higher difficulty Information processing speed Think fast 8 levels Number of correct responses in 60 s The system shows an image which must be sorted into the correct category as fast as possible. The number of categories and their similarity change with difficulty Numbers 8 levels Number of correct responses in 60 s The system displays a series of numbers that are randomly placed across the screen. The participant’s task is to tap the numbers in ascending order, as fast as possible Short-term memory Treasure hunt 8 levels Mean percentage of correct grid–square connections over 6 rounds The system shows a grid with a path to an “X” (treasure). The participant is asked to memorize the exact path to the treasure. The participant is then asked to reconstruct the path on a blank grid. Grid size, path length, and intermission time (time between memorization and reconstruction phase) are increased with higher difficulty Mental flexibility Mixer 8 levels Number of correct responses in 60 s The system displays a series of letters and numbers that are randomly placed across the screen. The participant must tap the numbers and letters in ascending/alphabetical order, as fast as possible, always alternating between the two Psychomotor speed Catch-a-cloud 6 levels Number of correct responses in 50 s (Test Mode: 20 s, Game Mode: 30 s) The system displays clouds on screen which must be “popped” by tapping on them as fast as possible. Once popped, the cloud reappears at another location on the screen. The first 20 s are the Test Mode (TM), which only shows 1 cloud on screen. After the TM, the Game Mode (GM) follows for 30 s. The GM includes gamification elements such as raining clouds, distractions (Balloons) which must be avoided, and multiple clouds at once Fig. 1 Screenshots of the challenges included in CoGames. *From left to right: Match Maker, Think Fast, Numbers, Treasure Hunt, Mixer, Catch-a-Cloud

Table 2 Domains, measures, and description of CoGames Cognitive domain Game (no. of levels) Primary measure Description Working memory Match maker 8 levels Number of correct responses in 60 s The system shows an either colorful or gray shape on the screen. With every answer, a new shape appears and the previous is hidden. The task is to continuously decide (“Yes”/”No”) whether the presented shape matches the one shown “n” taps before. “N” increases with higher difficulty Information processing speed Think fast 8 levels Number of correct responses in 60 s The system shows an image which must be sorted into the correct category as fast as possible. The number of categories and their similarity change with difficulty Numbers 8 levels Number of correct responses in 60 s The system displays a series of numbers that are randomly placed across the screen. The participant’s task is to tap the numbers in ascending order, as fast as possible Short-term memory Treasure hunt 8 levels Mean percentage of correct grid–square connections over 6 rounds The system shows a grid with a path to an “X” (treasure). The participant is asked to memorize the exact path to the treasure. The participant is then asked to reconstruct the path on a blank grid. Grid size, path length, and intermission time (time between memorization and reconstruction phase) are increased with higher difficulty Mental flexibility Mixer 8 levels Number of correct responses in 60 s The system displays a series of letters and numbers that are randomly placed across the screen. The participant must tap the numbers and letters in ascending/alphabetical order, as fast as possible, always alternating between the two Psychomotor speed Catch-a-cloud 6 levels Number of correct responses in 50 s (Test Mode: 20 s, Game Mode: 30 s) The system displays clouds on screen which must be “popped” by tapping on them as fast as possible. Once popped, the cloud reappears at another location on the screen. The first 20 s are the Test Mode (TM), which only shows 1 cloud on screen. After the TM, the Game Mode (GM) follows for 30 s. The GM includes gamification elements such as raining clouds, distractions (Balloons) which must be avoided, and multiple clouds at once

Domains, measures, and description of CoGames

Match maker

8 levels

Think fast

8 levels

Numbers

8 levels

Treasure hunt

8 levels

Mixer

8 levels

Catch-a-cloud

6 levels

Number of correct responses in 50 s

(Test Mode: 20 s,

Game Mode: 30 s)

Screenshots of the challenges included in CoGames. *From left to right: Match Maker, Think Fast, Numbers, Treasure Hunt, Mixer, Catch-a-Cloud

The acceptance questionnaire comprised five questions which were to be rated on a 5-point Likert scale (1 = most negative, to 5 = most positive): enjoyment, representation of mental performance, adequacy of difficulty increase, clarity of the instructions, and level of frustration. The full questionnaire is shown in the supplementary material (S1. Acceptance questionnaire).

Objectives and outcomes

Our objectives were to a) investigate the relative differences between difficulty levels; b) assess reliability; c) estimate potential practice effects and gain first evidence that the CoGames can detect change in performance over time, and d) assess acceptance by participants with a focus on enjoyment. Hence, our outcomes were a) disparities in linear mixed model estimates between levels, corrected for age, operating systems, education, and their interactions with difficulty levels. Each estimated parameter directly represents the difference in the mean performance outcome (e.g., number of correct responses in 60 s) between the level and the reference level. The further the estimated parameter is from zero, the greater is the difference; b) Spearman’s correlation coefficient between two repetitions of the same game and difficulty level, played on the same day. A correlation coefficient of r  ≥ 0.6 was regarded as acceptable; c) increase of scores across four repetitions of the level “Beginner” as estimate of practice effects and ability to detect change; d) mean ratings over the three rating days on the acceptance questionnaire. A score of ≥ 3 on the 5-point Likert scale was defined as sufficiently accepted by participants.

Data collection, statistical analyses, data access, and availability statement

Data collection, data cleaning, and feature extraction processes were performed using Python 3.9 coding language. C. L. and Ó. R. take full responsibility of these processes. Statistical analyses were performed using R version 4.4.0 (24.04.2024). L. K., S. P., and A. W. take full responsibility for the data, analyses and interpretation, and the conduct of the research and had full access to all the data and the right to publish any data separate and apart from any sponsor. The data supporting the findings of this study are available from the corresponding author upon reasonable request.

Results

We recruited 82 participants, 6 of whom dropped out, resulting in a final cohort of 76 participants. Dropouts were caused by technical difficulties ( n  = 3): crashes of the app which led to missing data and non-adherence ( n  = 3); inability to play the games within the fixed time frame foreseen for completion resulting in missing data. In total, a participant was scheduled to complete each game 22 times over 11 days including the 3 repetitions of level 1 (every game 2 × per day). The average overall adherence rate was 90.63%. Table 3 provides an overview of the participants’ demographics. Table 3 Demographics of participants Healthy volunteers ( n  = 76) Female, n (%) 51 (67%) Mean age ± SD Range 40.3 ± 13.9 years 18–69 years Adherence (tests completed of total scheduled) 90.63% Operating system, n (%), models iOS: 44 (58%), 16 different models between iPhone 7 and iPhone14 Android: 32 (42%), 20 different models from Samsung, OnePlus, Oppo, Xiaomi, and Sony Mean years of education ± SD Range 15 ± 2.5 years 10–20 years

Table 3 Demographics of participants Healthy volunteers ( n  = 76) Female, n (%) 51 (67%) Mean age ± SD Range 40.3 ± 13.9 years 18–69 years Adherence (tests completed of total scheduled) 90.63% Operating system, n (%), models iOS: 44 (58%), 16 different models between iPhone 7 and iPhone14 Android: 32 (42%), 20 different models from Samsung, OnePlus, Oppo, Xiaomi, and Sony Mean years of education ± SD Range 15 ± 2.5 years 10–20 years

Demographics of participants

Mean age ± SD

Range

40.3 ± 13.9 years

18–69 years

iOS: 44 (58%), 16 different models between iPhone 7 and iPhone14

Android: 32 (42%), 20 different models from Samsung, OnePlus, Oppo, Xiaomi, and Sony

Mean years of education ± SD

Range

15 ± 2.5 years

10–20 years

Outcomes

Difficulty level estimation

For all games, the level “Beginner” of day 1 was used as the reference. Detailed outputs of the linear mixed models for every game can be found in the appendix Tables 5 , 6 , 7 , 8 , 9 , 10 , 11 . Female and male participants did not show differences in performance. Therefore, we did not include this variable into the linear mixed models. In Match Maker the average number of correct responses in 60 s differed notably between all levels and the reference. Also, participants using iOS performed better (estimate for iOS vs. android: 16.67, 95%-CI [12.45, 20.88], p  ≤ 0.001), while older participants (estimate: −0.45 per years of age, 95%-CI [−0.59, −0.30], p  ≤ 0.001) performed worse. In Think Fast, the mean number of correct responses in 60 s of all levels, except for level “Apprentice” was different from the reference. Furthermore, younger age was associated with better scores (estimate: −1.25 per years of age, 95%-CI [−1.51, −0.99], p  ≤ 0.001). In Numbers, whereas the average number of correct responses in 60 s of the levels “Apprentice” and “Graduate” was not notably different from the reference, the measures of all other levels were. Younger (estimate: −0.23 per years of age, 95%-CI [−0.36, −0.11], p  ≤ 0.001) and participants with more years of education (estimate: 0.86 per years of education, 95%-CI [0.27, 1.46], p = 0.006) showed higher performance. For Treasure Hunt, only the measures (mean percentage of correct connections) in levels “Apprentice”, “Graduate”, and “Superstar” did not show considerable differences to the reference. In Mixer, the average number of correct responses in 60 s was substantially different between all levels, except for level “Graduate”. Higher performance was also associated with younger age (estimate: −0.29 per years of age, 95%-CI [−0.44, −0.16], p  ≤ 0.001) and more years of education (estimate: 0.71 per years of education, 95%-CI [0.04, 1.39], p  ≤ 0.001,). In Catch-a-Cloud Test Mode the average number of correct responses in 20 s did not differ strongly between levels. Younger participants (estimate: −0.24 per years of age, 95%-CI [−0.31, −0.17], p  ≤ 0.001,) and iOS users (estimate: 5.13, 95%-CI [3.12, 7.10], p  ≤ 0.001) performed better. For the Catch-a-Cloud Game Mode the average number of correct responses in 30 s of level “Superstar” clearly differed from the reference. Performance was better in younger (estimate: −0.33 per years of age, 95%-CI [−0.43, −0.23], p  ≤ 0.001,) and iOS users (estimate: 7.70, 95%-CI [5.46, 9.95], p  ≤ 0.001).

Reliability testing: test–retest correlation of the daily repetitions

Sample sizes for reliability calculations vary slightly due to single missing repetitions. The exact sample sizes used for each day are shown in detailed tables in the appendix (Tables 12 , 13 , 14 , 15 , 16 , 17 ): Achieved scores and rho (test–retest reliability) by study day and difficulty level. The three easiest levels (“Beginner”, “Apprentice”, and “Graduate”) of the game Treasure Hunt showed strong ceiling effects (see Table  15 : Treasure Hunt) and did not reach the target correlation coefficient of | r s | ≥ 0.6. All other difficulty levels of all games showed significant strong Spearman’s correlation coefficients (| r s | = 0.64 – 0.94, p  ≤ 0.001) in the comparison of test and retest, thus meeting the predefined target of | r s | ≥ 0.6: Match Maker: Med: 0.88, IQR: 0.77 – 0.91, range: 0.73–0.93; Think Fast: Med: 0.83, IQR: 0.8–0.91, range: 0.74–0.92; Numbers: Med: 0.78, IQR: 0.72–0.81, range: 0.66–0.85; Mixer: Med: 0.82, IQR: 0.78–0.87, range: 0.76–0.92; Catch-a-Cloud: Med: 0.9, IQR: 0.86–0.93, range = 0.84 –0.94. Treasure Hunt, including the levels that did not reach the target: Med: 0.55, QR: 0.32–0.72, range: 0.21–0.74. An overview of the correlation coefficients is shown in Table  4 . Table 4 Cognitive domains, primary measures, reliability, and acceptance (enjoyment) Cognitive domain Game Primary measure Reliability* (Spearman’s rho) Median, IQR Range Mean enjoyment rating (5-point Likert scale) Working memory Match maker Number of correct responses 0.88 0.77–0.91 0.73–0.93 3.8 (± 0.96) Processing speed Think fast Number of correct responses 0.83 0.8–0.91 0.74–0.92 4.28 (± 0.74) Numbers Number of correct responses 0.78 0.72–0.81 0.66–0.85 4.14 (± 0.85) Short-term memory Treasure hunt Mean percentage of correct connections 0.55 0.32–0.72 0.21–0.74 3.13 (± 1.19) Mental flexibility Mixer Number of correct responses 0.82 0.78–0.87 0.76–0.92 3.74 (± 1.01) Psychomotor speed Catch-a-cloud Number of correct responses 0.9 0.86–0.93 0.84–0.94 4.22 (± 1) * Correlation coefficients include all difficulty levels All p values of the correlations were ≤ 0.001 for all except the repetitions of level 1 (“Beginner”) in Treasure Hunt

Table 4 Cognitive domains, primary measures, reliability, and acceptance (enjoyment) Cognitive domain Game Primary measure Reliability* (Spearman’s rho) Median, IQR Range Mean enjoyment rating (5-point Likert scale) Working memory Match maker Number of correct responses 0.88 0.77–0.91 0.73–0.93 3.8 (± 0.96) Processing speed Think fast Number of correct responses 0.83 0.8–0.91 0.74–0.92 4.28 (± 0.74) Numbers Number of correct responses 0.78 0.72–0.81 0.66–0.85 4.14 (± 0.85) Short-term memory Treasure hunt Mean percentage of correct connections 0.55 0.32–0.72 0.21–0.74 3.13 (± 1.19) Mental flexibility Mixer Number of correct responses 0.82 0.78–0.87 0.76–0.92 3.74 (± 1.01) Psychomotor speed Catch-a-cloud Number of correct responses 0.9 0.86–0.93 0.84–0.94 4.22 (± 1) * Correlation coefficients include all difficulty levels All p values of the correlations were ≤ 0.001 for all except the repetitions of level 1 (“Beginner”) in Treasure Hunt

Cognitive domains, primary measures, reliability, and acceptance (enjoyment)

0.88

0.77–0.91

0.73–0.93

0.83

0.8–0.91

0.74–0.92

0.78

0.72–0.81

0.66–0.85

0.55

0.32–0.72

0.21–0.74

0.82

0.78–0.87

0.76–0.92

0.9

0.86–0.93

0.84–0.94

* Correlation coefficients include all difficulty levels

All p values of the correlations were ≤ 0.001 for all except the repetitions of level 1 (“Beginner”) in Treasure Hunt

Change in performance over time

For most games, there were gradual improvements over time when comparing the mean scores of level 1 (“Beginner”) that was completed on days 1, 2, 6, and 11. In Match Maker the mean score increased from 62 to 69, to 75 and then decreased to 73 on the last day. In Think Fast the score gradually increased from 86 to 103, to 112 to 116. The Numbers scores changed from 54 to 55 to 56 and remained at 56 for the last repetition. For Treasure Hunt the scores increased gradually from 96 to 97, to 98 to 99. In Mixer the scores increased gradually from 51 to 56, to 59 to 60. Lastly, Catch-a-Cloud Beginner scores gradually increased from 50 to 51 to 53 where it stagnated for the last repetition. Details of the results can be found in Figs.  2 a–g and the appendix (Tables 12 , 13 , 14 , 15 , 16 , 17 ). Fig. 2 a Match Maker: performance over time. b Think Fast: performance over time. c Numbers: performance over time. d Treasure Hunt: performance over time. e Mixer: performance over time. f Catch-a-Cloud (Test Mode): performance over time. g Catch-a-Cloud (Game Mode): performance over time

a Match Maker: performance over time. b Think Fast: performance over time. c Numbers: performance over time. d Treasure Hunt: performance over time. e Mixer: performance over time. f Catch-a-Cloud (Test Mode): performance over time. g Catch-a-Cloud (Game Mode): performance over time

Acceptance: Enjoyment

The average rating of enjoyment over all rating time points and across all games was 3.8 (range: 3.13–4.29) on a 5-point Likert scale: Match Maker: 3.8 (± 0.96), Think Fast: 4.28 (± 0.74), Numbers: 4.14 (± 0.85), Treasure Hunt: 3.13 (± 1.19), Mixer: 3.74 (± 1.01), Catch-a-Cloud: 4.22 (± 1). All games reached the predefined cutoff of 3 points on the 5-point Likert scale. Clear changes in ratings over time (> 0.5 points) were only found in Treasure Hunt: day 4: 3.47 (± 1.15), day 8: 2.78 (± 1.23), and day 11: 3.13 (± 1.19). An overview of the mean enjoyment ratings of all games is shown in Table  4 .

Acceptance: representation of cognitive function, perceived difficulty increase, clearness of instructions, and frustration

Across all games, the mean rating on whether the participants find that game performance represents their cognitive abilities was 3.73 (range: 3.26—3.94): Match Maker: 3.88 (± 0.92), Think Fast: 3.94 (± 0.83), Numbers: 3.83 (± 0.92), Treasure Hunt: 3.6 (± 1.15), Mixer: 3.87 (± 0.97), Catch-a-Cloud: 3.26 (± 1.3). There were no strong changes between the three rating days. On average, the participants rated the linearity of difficulty increase as 4 (range: 3.22–4.24): Match Maker was 4.23 (± 0.93), for Think Fast 4.15 (± 0.95), for Numbers 4.12 (± 0.92), for Treasure Hunt 4.20 (± 0.97), for Mixer 4.09 (± 0.9), and for Catch-a-Cloud 3.21 (± 1.24). The ratings increased over time for all games but Catch-a-Cloud, where the rating of the last day decreased: day 1: 2.9 (± 1.35), day 8: 3.42 (± 1.15), and day 11: 3.34 (± 1.12). Mean ratings on the clearness of the instructions were 4.79 (range: 4.67—4.86): Match Maker: 4.67 (± 0.72), Think Fast: 4.82 (± 0.54), Numbers: 4.84 (± 0.5), Treasure Hunt: 4.86 (± 0.41), Mixer: 4.70 (± 0.68), Catch-a-Cloud: 4.84 (± 0.54). Ratings regarding instructions did not change strongly over time. The average rating of the level of frustration was 3.9 (range: 2.83—4.72): Match Maker: 3.6 (± 1.07), Think Fast: 4.18 (± 0.91), Numbers: 4.23 (± 0.87), Treasure Hunt: 2.86 (± 1.29), Mixer: 3.84 (± 1), Catch-a-Cloud: 4.71 (± 0.65). For Match Maker and Treasure Hunt, the frustration rating between day 4 and day 8 decreased by 0.52 and 1.24 points, respectively. The detailed ratings for each day of questionnaire completion are shown in the appendix (Table  18 : Questionnaire ratings).

Discussion

In this study, we investigated the relative difficulty between levels, reliability, practice effects, and ability to measure change in performance over time, as well as the acceptance by participants of CoGames—a smartphone-based application for cognitive games that was developed in-house.

Overall, we were able to quantify the difficulty differences between levels. In most cases, the difficulty increased as intended; however, this was not always the case. All difficulty levels of all games reached our predefined cutoffs for reliability and acceptance (enjoyment). Furthermore, the observed increase in game scores over time suggests the presence of a measurable practice effect. However, the ability to detect change in cognitive function over time has to be further investigated.

Difficulty level estimation

Using the linear mixed model estimates of every level, we were able to assess the relative difference in performance of each difficulty level compared to the reference (level “Beginner”). These linear mixed models included age, years of education, operating system, and their interactions with the difficulty levels as covariates. A lower level estimate indicated worse performance and consequently higher level difficulty. Overall, most, but not all levels were notably different from the level “Beginner” depending on the game. Especially, lower levels appeared to be similar in difficulty to level “Beginner” (see Table  5 , 6 , 7 , 8 , 9 , 10 , 11 in the appendix). We assume this effect is the result of using the easiest level as the reference. Furthermore, the impact of covariates (age, operating system, and years of education) also varied across games and levels. Interestingly, according to these linear mixed model estimates not all levels increased in difficulty as designed (Level 1 = easiest, Level 8 = most difficult). Of course, these results must be interpreted carefully considering the potential influence of practice effects. In any case, these models quantify the difficulty levels and therefore provide evidence enabling the arrangement of levels toward adaptive games with consistently increasing difficulty. Standardization allows for comparing scores across difficulty levels. This is necessary if we want to monitor performance over time or compare scores cross-sectionally using this difficulty-adjusting system. To this purpose, we propose to standardize the scores according to the following formula: \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$Standardized\; score = \frac{raw\;score - predicted\;score}{{\sqrt {total\;varaince}}}$$\end{document} S t a n d a r d i z e d s c o r e = r a w s c o r e - p r e d i c t e d s c o r e t o t a l v a r a i n c e Raw score: the score achieved in the game (e.g., number of successful responses in 60 s). Predicted score: the score based on the linear mixed model depending on the level, age, operating system, and years of education. Total variance: variance of the prediction + residual variance.

Raw score: the score achieved in the game (e.g., number of successful responses in 60 s).

Predicted score: the score based on the linear mixed model depending on the level, age, operating system, and years of education.

Total variance: variance of the prediction + residual variance.

Importantly, predicted values may vary in different populations.

Reliability

All difficulty levels across all games reached the predefined cutoff coefficient of | r s | ≥ 0.6 (range: 0.64—0.94), except for the repetitions of the three easiest difficulty levels (“Beginner”, “Apprentice”, and “Graduate”) of the visual short-term memory game Treasure Hunt. As shown in Table  15 , these three lowest levels had a clear ceiling effect with most participants achieving a close to perfect score (median > 90% correct answers). This range restriction results in reduced variability and consequently in a lower correlation coefficient. Supporting this interpretation, the correlation coefficient increased in the higher difficulty levels, in which we observed wider and less skewed score distributions (Table  15 ).

While the number of studies successfully proving high reliability of self-administered digital versions of established neuropsychological tests (e.g., SDMT) is large [ 11 , 14 , 28 – 32 ], the published evidence regarding the reliability of gamified smartphone-based cognitive tests is scarce. Nonetheless, the existing literature supports our findings: Brewster et al. (2021): ICC range = 0.85–0.87; Wiley et al. (2024): ICC = 0.58; Staffaroni et al. (2024): ICC range = 0.77–0.95 [ 33 – 35 ]. It is noteworthy that a direct comparison of the referenced studies with the results of CoGames is compromised by the different study designs (frequency, total duration, number of repetitions, etc.) and the timing of test and retest. Nevertheless, all studies suggest that smartphone-based tests can provide a reliable assessment of cognitive performance.

Change in performance over time

As expected, we found performance improvements in most games when comparing the scores of the easiest level that was repeated at days 1, 2, 6, and 11 [ 36 ]. As shown in Fig.  2 a–f, the performance of Match Maker, Think Fast, and Mixer showed either a linear increase over time or an initial steeper increase, followed by a plateauing of the curve. No major change was detected in the performance curves for Numbers, Treasure Hunt, and Catch-a-Cloud. As for Treasure Hunt, we attribute this to very low difficulty of the “Beginner” level which led to a ceiling effect. Once this ceiling (i.e., 100% correct responses) is reached, further improvement is impossible. Numbers and Catch-a-Cloud, are reaction time-based and less cognitively demanding games. Such tasks are easier automated and do not require as much practice until a plateau of maximal performance is reached compared to tasks that are more complex and cognitively challenging [ 37 ]. Overall, younger age correlated with better scores. However, the practice effect appeared to be similar in all age groups.

In any case, although not definitive proof, the fact by itself that these practice effects were observable suggests that CoGames might have the potential to detect performance changes over time. This is another mandatory prerequisite for a useful monitoring tool, but also an effect that must be considered in clinical monitoring.

Acceptance by participants: enjoyment

In recent years, awareness of the added value of including gamification elements in digital applications for rehabilitation, therapy, or assessment in MS has gained increasing interest [ 10 , 38 , 39 ]. Factors such as enjoyment, user-friendliness, and meaningfulness are fundamental to achieve high adherence to monitoring tools. In this study, we investigated elements of acceptance on a 5-point Likert scale (1 = worst, and 5 = best), with a focus on enjoyment. Our findings show that all games are at least moderately enjoyed (range: 3.13—4.22). We assume that gamification aspects, such as audible and visual feedback to correct and false responses, colorful design of interactive items and background, and points as rewards for good performance contribute strongly to the enjoyment factor, hence encouraging the acceptance and use of such tools.

In most of our games, the enjoyment rating stayed the same over the three time points (day 4, 8, and 11). Minor differences in enjoyment ratings across the three different rating days might indicate which difficulty levels were preferred: Lower enjoyment ratings during higher levels in the games Treasure Hunt and Match Maker reflect our results, which show that the difficulty increase in those two games was higher compared to the other games. This might have led to more frustration in higher levels of those games. In support of this interpretation, the “frustration” ratings showed similar changes at the same time of questionnaire completion as the enjoyment ratings (Table  18 : Questionnaire ratings).

Acceptance by participants: representation of cognitive function, perceived difficulty increase, clearness of instructions, and frustration

Participants perceived all games as representative of their cognitive abilities and found the difficulty increase well designed, meaning the levels seemed to get more difficult by the day (excluding the repetitions of level 1). The in-app instructions were rated as very clear across all games. Not surprisingly, the frustration ratings correlated strongly with the enjoyment ratings, which is especially visible in the ratings of the higher levels. These findings are comparable to results of our previous study [ 24 ].

Overall, the participants found the CoGames enjoyable, representative, and well designed regarding user friendliness, clarity of instructions, and difficulty level transitions, which is an excellent foundation for high adherence when using a monitoring tool.

Limitations

We cannot rule out a potential recruitment bias toward participants with a higher affinity for digital devices and gaming. The enjoyment rating might therefore be different in the general population. We attempted to minimize this possible bias by broad recruitment across different age groups not only via digital media (website), but also via flyers in the hospital and public places and offered a small remuneration for participation. The recruitment of healthy volunteers was purposely chosen to optimize reliability testing without disease-related influences. Consequently, the acceptance ratings need to be confirmed in PwMS of different disability levels. Monitoring of cognition using an adaptive difficulty system might also serve as cognitive training which is important to consider when interpreting the data. However, with an adequate frequency we expect the practice effect to be minimal once the individual performance level is reached. Our study population might not have been large enough to be representative of the general population. Therefore, our results need to be replicated in further larger studies with longer follow-up. Lastly, the short study duration did not allow an in-depth and conclusive analysis of practice effects and sensitivity to change.

Conclusion and future research

The results of this study support the hypothesis that adaptive, digital, and gamified cognitive tests can be used as a reliable and well-accepted assessment and monitoring tool. Besides improved convenience and enjoyment, CoGames aims to provide a more comprehensive assessment of cognitive function by covering multiple cognitive domains. Whether CoGames can assess specific cognitive domains as intended is further investigated in a currently running validation program, where the adaptive versions of CoGames are being compared both cross-sectionally and longitudinally to established neuropsychological tests in a population of PwMS and healthy volunteers (dreaMS Validation study 1, NCT05009160 ). Using the thresholds of the 25th and 75th percentile of every level found in the CoGames study, we implemented a dynamic difficulty adjustment (DDA) system, which allows patients to level up and down depending on their individual performance. This adaptive design is intended to promote the state of high concentration, keeping users engaged and motivated by avoiding frustration (task too difficult) and boredom (task too easy). Furthermore, the difficulty level quantification allows us to rearrange the levels for optimal and consistent difficulty increase in the next CoGames version. Lastly, we propose a standardization of the scores, enabling comparisons of performance across different difficulty levels.

Electronic supplementary material

Below is the link to the electronic supplementary material. Supplementary file1 (DOCX 13 kb) Supplementary file2 (MOV 11431 kb) Supplementary file3 (MOV 40343 kb) Supplementary file4 (MOV 10032 kb) Supplementary file5 (MOV 9088 kb) Supplementary file6 (MOV 12293 kb) Supplementary file7 (MOV 9706 kb)

Supplementary file1 (DOCX 13 kb)

Supplementary file2 (MOV 11431 kb)

Supplementary file3 (MOV 40343 kb)

Supplementary file4 (MOV 10032 kb)

Supplementary file5 (MOV 9088 kb)

Supplementary file6 (MOV 12293 kb)

Supplementary file7 (MOV 9706 kb)