Exercise Through Mobile
Chandrika Jayant and T. Scott Saponas
December 14, 2005
Obesity is a major health risk whose frequency is high in the United States. Physical activity has been shown to be an eﬀective way to combat obesity. We build on previous work of motivating physical activity socially and with entertainment by developing a system for playing video games while mobile and comparing game scores and step counts with friends.
Figure 1: Nintendo Super Mario Bros. game used in project
There are over 60 million obese people in the United States. Sixteen percent of young people aged six to nineteen are overweight. One of the many ways to address this major health risk is physical activity. Recent work suggests entertainment and social groups can be an eﬀective means for motivating phys-ical activity[3, 10]. About 60% of all Americans age 6 and older play computer or video games. 39% of interactive game players are 17 or under. Leveraging this national and international love of gaming for exercise could have a large positive impact on people’s health. We propose combining social and entertainment motivation through mobile videos games. We hy-pothesize video games that can be played while mobile and controlled by one’s body movements can be both entertaining and healthful. Speciﬁcally, if people are able to play video games while on the move and compare their game scores and step counts with that of their friends, we believe they will be more physically active. In this paper, we present MarioFit: a system for playing the 1985 Nintendo game Super Mario Bros (Figure 1) on a PDA using one’s body movements as the input mechanism. We utilize the University of Washington (UW) & Intel Research Seattle (IRS) MultiSensorBoard (MSB) to gather accelerometer and compass data. Based on this data we infer six human movements as input to the game: jumping, ducking, turning, walking, running, and throwing. MarioFit also includes a web site where users can login and compare their Mario scores and step counts with their friends (Figure 2). The rest of this paper is organized as follows: We begin with related work in the area of motion-driven video games and motivation for physical activity. In Section 3 we describe our approach to mobile human-movement driven games including our technical setup and inference algorithms. We then compare the PDA’s buttons to our input mechanism and evaluate our jumping classiﬁer (with two users) in Section 4. We end the paper with a discussion in Section 5 and a conclusion with future work in Section 6.
In this section, we describe related work in physical activity motivation using mobile technology and sensing motion as video game input. The arcade has seen a growing trend of games utilizing human movement as input. Get Up Move
Figure 2: MarioFit Online - compare scores and steps online
started in 2004 with the goal of promoting the beneﬁts of using dance video games for physical ﬁtness. Games like ParaParaParadise and Dance Dance Revolution(DDR) have been a world-wide craze the past few years. Many people have started using these dance video games as their primary source of exercise. Applications with body user interfaces are also becoming more popular–games in which a player uses his natural body movements for controlling the game. This seems to add to players sense of presence in the environment and could add to higher motivation levels with added realism. Human Pacman developed at Nanyang Technological University incorporates physical enactment of movements as input for their games, although in a more virtual setting. The user has wearable computing, a large hat, glasses, and a vest. This is another example in which we see a growing interest in physical gaming and entertainment. There is an increase in applications using accelerometers for detecting gestures such as tilt in hand held consumer electronics devices like cell phones and TV remotes. The Power book Apple Motion Sensor uses physical movement of the Apple Power book to allow the user to control activities such as scrolling and gaming without a mouse or keyboard. Nintendo is about to release its ”Nintendo Revolution,” which has a controller that senses motion, depth, positioning, and targeting as the user plays.. However, the users are meant to be mostly stationary except for their hands and arms. One of our contributions is to try to infer more movements like jumping and walking. The mentioned work discusses using sensors or physical human motion to drive video games using only accelerometers. We hypothesize that more types of video games could be used for exercise in addition to entertainment. However, users may want more motivation than just being entertained by the game. The users should know how much exercise they are getting from these games so they may set goals and see progress. Current accelerometer technology allows for relatively good speed and incline predictions. Usually more than one sensor must be worn, however (i.e. on the heel and on the back). In time more accurate measurements with a single sensor location may be possible. There are also more
Figure 3: User playing game with sensor board
complex devices for measuring physical activity with much higher accuracy, which can calculate duration, frequency, and intensity of diﬀerent activities, such as the Intelligent Device for Energy Expenditure and Activity (IDEEA). So far, these devices are still rather cumbersome and expensive (the IDEEA has 5 sensors connected via wire to a data collection device worn on the belt). Researchers at IRS and UW have investigated using ubiquitous computing for physical activity, in both the technological and motivational aspects. In particular, Everitt has studied the social factors present in gaming. In her paper, users share step counts (from pedometers) via their cell phones. After a 3 week in-situ study, some participants were motivated by competition and goals while others were motivated by collaboration. We believe that video games using physical human motion can be used in a similar way- showing step counts, displaying caloric output, and allowing data sharing or modes of competition with others via the mobile device or online. There have been other studies showing that children and adolescents can be motivated by computer and video games to change their self care and health behavior, in regards to diabetes, smoking, and asthma. If people can willingly improve their health using interactive games, it gives us hope that using mobile games for exercise could also be highly motivating.
We want to allow users to play games on their cell phones and PDA’s using their physical activity as input. We hypothesize that competition and social factors could motivate users to play these games. In this paper we tackle the ﬁrst problem of ﬁnding out whether this sort of application is usable. We use an accelerometer and compass to infer some human motion and use that as input to drive a Nintendo game on a PDA. In Figure 3 we see a user playing our game.
3.1 Technical Setup The MarioFit system is composed of a sensing platform (MSB) connected to a PDA that communicates to a web site. The overall architecture is shown in Figure 5. The MSB is used to collect the sensor data for detecting human movements. The MSB is built on a 6-layer PCB that includes an Atmel ATMega 128L microprocessor running at 7.3728MHz. The MSB is equipped with 7 sensors; we just use the compass and accelerometer in our project. The MSB is shown
Figure 4: the multi-modal sensor board and the HP iPAQ
in Figure 4. The 2-axis digital compass data is sampled at 30 Hz and the 3-axis acceleration is sampled at around 550 Hz. The compass sensor sends data to the ATMega 128L microprocessor via the I2C Communication Bus. The accelerometer uses the SPI Communication Bus. The sensor board is connected to the Hewlett-Packard iPAQ hx4705 Pocket PC (Figure 4). The UWAR annotation program runs on the iPAQ and collects sensor data from the MSB. We have this program buﬀer 30 MSB frames at a time then send them in the UWAR2 format over a localhost TCP socket connection (port 1567) to our Nintendo emulator. Our Nintendo emulator is based on the Pocket Nester Project, an open source Nintendo emulator for the PocketPC written using Embedded Visual Tools 3.0(,). We modiﬁed this emulator to receive the UWAR2 data stream on TCP port 1567. We then process the data as described in Section 3.2 and provide input to the emulator. We also provide MarioFit Online, a web site where users can login and see the scores and step counts of themselves and their friends. We have the emulator occasionally make an HTTP request to the web site with the users current high score for the day and step count. We aggregate this by day in a Microsoft SQL Server database. When users login we then dynamically generate Microsoft Excel graphs using this data and stream to the browser as an image.
3.2 Algorithms We came up with algorithms to discern diﬀerent inputs to the Nintendo emulator (see sample Mario moves in Figure 6). For this project we focused on walking, running, jumping, turning, throwing, and ducking. We had to ﬁnd a mapping from these gestures to the Nintendo buttons (A, B, right, left, up, and down - sometimes in combination). The users are told not to really pay much attention to the sensor in their hand, to try to act as naturally as possible. For walking, the user should just walk at a reasonable pace. To run, they should run, or just walk briskly. To jump, they should jump in the air with both legs or just one leg. To throw, they should throw with a brisk movement of the right wrist from left to right on the x-plane (parallel to the ground). To duck, they should duck low to the ground. And to turn, they should turn 180 degrees.
Figure 5: Overall Architecture of System
3.2.1 Walking, Running, & Jumping For walking, we look at only the z axis (perpendicular to the ground) acceleration from the 3-axis accelerometer. We also look at the acceleration on this axis for jumping. For both of these actions we look for only positive z acceleration (to help diﬀerentiate between ducking and jumping/walking). There is a threshold for jumping; once the z acceleration passes this point within a period of time we say the user has jumped. If the z acceleration is under this jumping threshold but over some lower positive acceleration walking threshold, we say the user is walking. We calculate approximate number of steps by putting a limit on approximately how many steps can be made within a certain period of time. Diﬀerentiating running and walking is as simple as just looking for a running threshold number of steps per unit time, which we set at about double that of walking. This seems to work well in practice. When the accelerometer streams data which we interpret as running, we tell the Nintendo emulator to press down another button (B) in addition to the right button (for walking) to make Mario run.
3.2.2 Turning For turning, the 2-axis digital compass is used. When a user turns more than 180 degrees in any direction, Mario will change directions. We only use relational direction- absolute direction does not matter. If we used a smaller angle to signal turning, this would disallow users to walk around ”casually” and make slight turns, or turn corners, which we assume would not be intended to be actual turns in the game. Problems can occur with the compass when the user is around structures with lots of metal or when the calibration is bad.
3.2.3 Throwing Throwing uses x axis acceleration. To throw, the user must move the sensor from left to right (or right to left), just as long as the movement is quick and relatively parallel to the ground.
3.2.4 Ducking The last gesture we implemented was ducking. We look for negative z acceleration below a certain threshold. Ducking is diﬀerentiated from the downward part of jumping by putting a delay after a jump is detected, similarly with walking. Once the user ducks, we make sure in the emulator that the ducking
Figure 6: sample Mario moves- jumping and throwing
User A B
MarioFit 600 870
Traditional PDA 1800 14000
Table 1: Mean Mario scores over three games with each input method
motion is held for a signiﬁcant enough period of time so Mario doesn’t just hop down and then pop back up quickly.
In this section we describe our evaluation of our input method and jump classiﬁer.
4.1 Experimental Methodology We conducted a within-subjects comparison of our input method and PDA buttons as input mechanisms for Super Mario Bros. We recruited two computer science graduate students (one male and one female) to play Super Mario Bros on an iPAQ PDA. Both participants were experienced in video games. After a brief explanation and practice game, each participant played the game three times using each input method. The order of the input methods was varied (counterbalanced) between users. We video taped participants when they played using our input method. The MarioFit system logged each time one of the six human movements was detected by the system. Participants’ Mario scores were also recorded. We would like to give analysis of all of our classiﬁers. However, in our experiment participants walked almost the whole time they played and performed few other movements beside jumping. For this reason the only classiﬁer we analyze is the jump classiﬁer. To gather ground truth for jumping, we hand-labeled the video data for instances of the user jumping.
4.2 Results Both participants had a better mean Mario score using the PDA’s buttons for input (Table 1). User A’s diﬀerence in mean Mario score was much smaller than User B’s. This suggests one’s ability to play Mario
User A B
Precision 60.71% 70.42%
Recall 77.27% 81.97%
Table 2: Precision & recall of jumping classiﬁer
Figure 7: User A - Jump Classiﬁer
with the PDA’s buttons has little eﬀect on ability to play with our input method. Our users said they had fun while playing the game with our input method. One user said they worked up a sweat because of the physical activity involved. When using the PDA buttons as input one user leaned against a wall. The other participant sat down while they played using the buttons. Both users chose to take oﬀ their long-sleeve upper layer while playing the version with our input method. Both participants commented it was more diﬃcult to play with our input method than using the PDA buttons. Figure 7 shows classiﬁcation and ground truth for User A’s jumping. The horizontal axis shows the time in seconds when the instance of jumping occured. If both ”Ground Truth” and ”Classiﬁer” are present at a time instant then classiﬁcation is correct. In other cases either the classiﬁer missed an instance of jumping or it incorrectly detected jumping. Figure 8 shows the same information for User B. From this data we calculated the precision and recall for our classiﬁer for each user (Table 2). The jump classiﬁer had higher precision and recall for User B. The jumping classiﬁer had a higher recall than precision for both users.
Figure 8: User B - Jump Classiﬁer
Our results show it is possible to play Super Mario Bros and potentially other games using human movements as the input mechanism. With our implementation it is not as easy as using the buttons on the PDA. However, it is still unknown whether this input mechanism aﬀects the entertainment level of the game. It is also unknown whether this game is entertaining enough to motivate more physical activity. Based on the success of other motion games like DDR and the popularity of Super Mario Bros, it is plausible MarioFit could be motivating for some individuals. Interest in the game itself exists: it has sold approximately 40 million copies in North America alone. However, combining the game with exercise could be another story. While our users did not score well using our input method, it is hard to say whether low scores are due to this input mechanism being diﬃcult, diﬀerent from traditional input mechanisms, or both. This could be further investigated by having users play games using this input method over an extended period of time. Alternatively, one could run a similar experiment to ours with people who have very little video game experience. The jumping classiﬁer’s recall was higher than its precision for both users. We do not know whether this good or bad. It would be interesting to investigate if it is worse when the user performs a movement that is not recognized or when a user is walking and not performing other movements like jumping and the character jumps. One could begin to investigate what is more important by varying the sensitivity of the classiﬁer and having subjects play the game using diﬀerent sensitivities and compare their scoring with diﬀerent sensitivities. Our evaluation is limited in its generality both by our small number of participants and by the short period of time people played. While our participants said it was fun to play, there could be novelty eﬀects and it would be interesting to know if it is still fun to play with our input method when it is no longer novel to the user. A longitudinal in-situ study would also help provide insight into whether it
helps motivate physical activity. Our implementation is also limited by our sensing platform (MSB) and our choice of a web page to communicate the scores of user’s friends. We found our compass sensor to often be inaccurate. It suﬀered from both calibration problems and sensitivity to large metal objects and structures. Our system can only communicate scores and step counts when the iPAQ has an Internet connection. Users can only see their previous scores and step counts as well as their friends when they have an Internet connection and web browser (on the PDA or otherwise). Using human movements as an input mechanism could have negative social eﬀects. For example, are the required movements too obtrusive in normal settings? Would one be embarrassed to perform them in public? If the answer to these questions is yes then people might ”cheat” and move the sensor with their hands to emulate the ”real” movements resulting in a more ”lazy” game experience.
Conclusion & Future Work
Physical activity can be an important component of health. In this paper we presented MarioFit, a mobile gaming system where people use their body movements to control the video game. Users are able to track their step count and game scores and compare them with their friends. We hypothesize providing such a social gaming system can motivate some people to be more physically active. Evaluation of this system as a motivation tool is the next step. A longitudinal in-situ user study would allow one to investigate the extent to which people are motivated by entertainment, competition, and collaboration. In the future we would want to explore reporting caloric output or other measures of physical activity in addition to users’ step count. We would also want to conduct variations on our experiment with subjects trying to walk somewhere and use diﬀerent input mechanisms. While we chose Super Mario Bros as the game for people to play, we would also want to explore creating new games designed just for physical activities that use sensing capabilities like the MSB. For example, one could imagine a game where you are competing with another person (communicating over a cell network) to get to places with certain properties. For example, the game might say whomever gets to get to a place high enough to have a certain air pressure gets 500 points. It would be interesting to investigate whether games like this could be as entertaining as traditional games yet provide even greater physical activity. Obviously, we would want to design these games to encourage the use of one’s body and not motor vehicles.
 K. Everitt. Motivating physical activity using social support through mobile phones.
 Fitness and Kids: http://www.ﬁtnessandkids.com/art Dance Dance-revolution.html. How dance dance revolution is getting kids oﬀ the couch.
 Center for Disease Control: http://www.cdc.gov/nccdphp/publications/aag/dnpa.htm. Physical ac-tivity and good nutrition: Essential elements to prevent chronic diseases and obesity.
 R. Herren, A. Sparti, K. Aminian, and Y. Schutz. The prediction of speed and incline in outdoor running in humans using accelerometry. July 1999.
 Nanyang Technological University: infor.html. Human pacman.
http://126.96.36.199/research/HP/HP webpage/research HP-
 Center For Disease Control: http://www.cdc.gov/nccdphp/dnpa/obesity. Overweight and obesity.
 B. Dybwad http://www.engadget.com/entry/1234000680037005/. Powerbook apple motion sensor as game controller.
 http://www.getupmove.com. Get up move: Promoting ﬁtness, conﬁdence, and fun with video games.
 J. DeGaspari http://www.memagazine.org/backissues/nov01/features/rattle/rattle.html. Shake rat-tle and roll.
 http://www.nintendo.com/newsarticle?articleid=TfyOgnUc7zB ZlzdQvMhKJOgS5LsR2nK&page=newsmain. Nintendo reveals ’revolution’-ary controller in keynote speech.
 http://www.nintendoland.com/home2.htm?funfacts/stat.htm. records.
 J. Lester, T.Choudhury, G. Borriello, S. Consolvo, J. Landay, K. Everitt, and I. Smith. Sensing and modeling activities to support physical ﬁtness. InUbiComp ’05 Workshop: UbiComp to Support Physical Fitness, Tokyo,Japan, 2005.
 D. Lieberman. Management of chronic pediatric diseases with interactive health games: Theory and research ﬁndings. 2005.
 S. Mokka, A. Vaatanen, J. Heinila, and P. Valkkynen. Fitness computer game with a bodily user interface. InProceedings of the 2nd International Conference on Entertainment computing ICEC ’03, Pittsburgh,PA.
 K. Zhang, P. Werner, M. Sun, F. Pi-Sunyer, and C. Boozer. Measurement of human daily physical activity. January 2003.
All of our appendices can be accessed athttp://www.cs.washington.edu/homes/ssaponas/cse567/ appendices.html.
7.1 Videos Two videos of MarioFit in use can be found athttp://www.cs.washington.edu/homes/ssaponas/ cse567/index.html.
7.2 Source Code The source code for MarioFit is available athttp://www.cs.washington.edu/homes/ssaponas/cse567/ src.html.