PERSONAL ACTIVITY MONITORING ... factors in real-time; carefully looking into the monitoring ..... system compares the time-window at the missed/rejected.
AN INTELLIGENT NOTIFICATION SYSTEM USING CONTEXT FROM REAL-TIME PERSONAL ACTIVITY MONITORING Hyungik Oh, Laleh Jalali, Ramesh Jain Department of Computer Science, University of California, Irvine {hyungiko, lalehj, jain}@ics.uci.edu ABSTRACT People can now receive custom-made information through smartphones, tablets or wearable devices. However, people often tend to miss vital information, even reminders, in the flood of notifications. The problem of finding convenient moments for need-to-know information should be investigated. Because each person’s message awareness pattern on a smart medium might be different, the necessity of personalized notification time should be emphasized. We believe that tracking changes in a user’s physical activity and other contextual factors will reveal the most convenient moments. We propose a mobile framework, smartNoti, to carefully examine the user environment. The main contributions of our framework are: 1) developing an architecture to provide crucial information in a timely manner at a recognizable moment; 2) integrating, processing, training, and storing personalized latent features from heterogeneous data streams; 3) detecting user context transitions that might provide recognizable and available moments; and 4) predicting these moment and providing a notification message. The experimental validation on Intelligent Callback Reminder, which we implemented on an android application to notify a user missed or rejected call, demonstrates that our approach is effective. We believe that our findings can lead to intelligent strategies to issue unobtrusive notifications on today’s smart phones at no extra cost, by using sensors and contextual factors. Index Terms— Notification system, context awareness, contexture modeling, user logging, moment prediction 1. INTRODUCTION The Intellectualization of the notification systems is a principal challenge in the Datafication [1] world. The emergence of so-called cyber-physical and cyber-physicalsocial systems [2] augments the necessity of suitable notification times. For example, the online mass media pushes breaking news to application software users. Social network systems notify users whenever updated events occur. Calendar events send alarms at the planned time. Even fitness programs press users to achieve their daily
goals. On one hand, these notification messages can be helpful, on the other hand, people might soon be overwhelmed with the huge amount of notifications. To avoid the deluge of information at the involuntary times, notification systems need to be made aware of their current environment. Infrastructures for understanding user environments, such as wearable sensors, smart phones, and network systems have grown in use. Therefore, now is an ideal moment to explore and design the timeliness in notification systems. The challenges of context-base notification systems are: selecting fundamental context factors that influence the activity patterns of individual agents; keeping track of these factors in real-time; carefully looking into the monitoring data; and immediately coping with recognized desirable situations. Considering these challenges, we propose a mobile framework, smartNoti, to provide a standardized method for finding the recognizable and available moment. “Recognizable moment” signifies a time to be noticed by the user. “Available moment” means a time that the user can carry out the suggestions made by notification message. We propose the hypothesis that some appropriate moments will exist at context transition moments. We define context in the same way as that A.K Dey et al. defined it in [3], apply the cybernetic principles [2] into the mobile framework, and then suggest a method to predict the moment. To demonstrate our approach, we implement a mobile notification application called Intelligent Callback Reminder on smartNoti. The applicability of this approach is demonstrated on a case study of the smartNoti system focusing on 6 users over a period of about two months. The application details are as follow: Purpose • To notify the user of callback reminder messages in the case of missed/rejected calls at the recognizable and available moment. Scenario • smartNoti keeps track of user contexts. When a missed/rejected call occurs, smartNoti starts detecting context transitions based on real-time user logging. For
example, a user receives a missed call during a group meeting. After the meeting, the context transition between meeting and chatting is caught by smartNoti. The system judges whether the moment is recognizable and available for the notification. If it is an ideal moment, smartNoti notifies the user with callback reminder message. If it is not an appropriate time, smartNoti keeps detection process running. The main goals of our proposed framework are: to provide an architecture to notify crucial information in a timely manner at a recognizable moment; (i) integrate, process, train, and store personalized latent features from heterogeneous data streams; (ii) detect context transitions that might constitute recognizable and available moments; (iii) predict these moment in a timely fashion; (iv) and provide notification messages. This line of research is important because there are large amounts of messages trying to get in touch with users. To find the best notification moments, we assume that individuals have their own preferred moments to receive notifications and these will be one of context transition moments. 2. RELATED WORK Research on notification systems that investigates appropriate moments has become popular. However, to the best of our knowledge, there is no specialized mobile framework to find both recognizable and available moments through a total directional approach – real time chronological observation and analysis of user contexts, and a statistical model. Current studies try to minimize user interruption by choosing the best moments. While there are many factors that influence a person's interruptibility, the literature has found current user activities to be the most relevant. Interruptions between coarse breakpoints, i.e., major changes in workflow, produce less annoyance, and users have assessed them as being more respectful of their ongoing activity than random moments [7, 8]. To detect and differentiate between these breakpoints in interactive tasks, models can be created. Ho and Intille [9] used two accelerometers and activity recognition to trigger interruptions at moments when users change the activity, e.g., when from sitting to walking. They showed that users are significantly more receptive to interruptions than a random condition. Kern and Schiele used wearable accelerometers, audio, and location sensors to decide whether a user should be notified and which modality to use. They argue that no advanced model is needed, as the combination of tendencies is already sufficient [10]. 3. ARCHITECTURE Figure 1 shows the architecture of smartNoti. By following the cybernetic principles [2], building models, measuring parameters, estimating states, and controlling the system, we
Fig. 1. Architecture of smartNoti system.
divide our architecture into three layers: logging, processing, and prediction. Logging layer - Measuring parameter. The logging layer contributes to two main tasks: context factor logging and event factor logging. Context factor logging integrates six different categories: activity (physical activities and number of steps), location (latitude, longitude, and venue), time (week, weekend, or dawn, morning, afternoon, evening), logical (calendar), entertainment (audio, video, application usage), and system (silence, vibration, bell). More categories can be dynamically added as needed. Some examples of event factors are incoming/outgoing calls or missed/rejected calls. Processing layer - Building models. Context factors are processed every 5 minutes in this layer. Processed data is used to build a context matrix the time-window. Whenever event factors occur, the processing layer builds the user model using time-window at that point. Time-window is stored in Temporary Data for only 5 minutes. If event factors occur within this time, Model Learner uses the stored time-window and trains the model. The stored time-window is updated every 5 minutes. The Naïve Bayesian Model is maintained in a local path as a type of SQLite database with necessary data such as average, standard deviation, and the sum of squares of differences from the (current) mean (m2). The model does not keep all of its index data, but rather uses the online algorithm of [5,6], and updates incrementally to calculate the posterior probability. Prediction layer 1 - Estimating states. Context-ChangeDetector (C-C-D) estimates the current state by calculating the distance between target time-window and incoming time-window. “Target” means the time when event factors occur, and “incoming” means every 5 minutes. We estimate whether the incoming user context has changed compared to the target context. The estimation algorithm is described in Algorithm 1.
Prediction layer 2 – Rule base filtering. smartNoti passes the incoming matrix through a pre-defined rule filter, and handles each context by following four different rules: release, skip, direct notification, and maintenance. Release means to stop tracking the moment, skip signifies to ignore the current context and wait for new incoming. Direct notification means to immediately notify, and maintenance means to keep tracking the moment. Each category can easily include additional rules. Prediction layer 3 - Controlling the system. If C-C-D detects context transition moment, the Moment Predictor predicts whether the detected time is the recognizable and available moment, and then generates notifications. 4. CONTEXT CHANGE DETECTION Chronological observation and analysis of user context gives insight into which moments should be focused on. We are confident that there must be a suitable notification moment between the i th context and the (i +1)th context. For example, when a person who has been working in an office moves to go to the restroom, we detect that transition and check the moment through a model. In particular, we try to figure out the most timely moment among all those of context transitions. The method for detecting context change is to generate time-windows first and then to catch the transition by calculating the distance between the target window and the incoming window. The most important aspects of calculating distance are: what kind of context factors should be utilized; how to simply but effectively compare two contexts; how to deal with a different range of Algorithm 1 Context Change Detection Algorithm Input: Ti = ( F1, F2 , F3,..., Fn ) ; // 1 ≤ i ≤ 288 , n = the number of context factors Output: TRUE, FALSE 1: if C-C-D is tracking context change 2: Calculate distance between Tk and Ti ; // Tk = Target time-window, // Ti = Incoming time-window 3: 4:
if distance > threshold return TRUE; // Detect context change, and stop tracking 5: else if maximum tracking time up 6: return TRUE; // Stop tracking 7: else i++; 8: return FALSE; // Wait next time-window, keep tracking 9: endif 10: else if Ti == tracking event factor 11:
Fig. 2. Generalized logistic curve of step counts. K = 4, A = 0, Q = 0.01, B = 0.01, M = 150, v = 0.01
Fig. 3. Generalized logistic curve of unique application counts. K = 2, A = 0, Q = 0.1, B = 0.5, M = 6, v = 0.1
continuous values and harmonize with discrete values; and how to personalize the transition threshold. This is described in the following sections and in Algorithm 1. 4.1. Latent features We divide the latent features into continuous variables and discrete variables. Continuous variables include activity level, step count, and amount of application usage. We define activity level as average intensity of accumulated physical activities. Each physical activity type, (standing still, tilting/unknown, on foot, on bicycle, in vehicle) is scored from 0 to 4 by its intensity; its frequency during five minutes is counted and then the level is calculated in (1) where i is the intensity of the activity and k is the unique number of accumulated physical activities.
Set Ti to target time-window Tk , start tracking;
12: i++; 13: return FALSE; 14: endif
4
∑i × (PhysicalActivity)
i
C1 =
i=0
k
(1)
Step count and the amount of application usage needs to be normalized. Figures 2 and 3 show the generalized logistic curve [4] of step count and the amount of unique application usage. By heuristically normalizing these factors with the generalized logistic function (2), we try to reasonably convert original values to normalized values. K−A Ci (t) = A + (2) (1+ Qe−B(t−M ) )1/v Discrete variables include media usage (true or false), sound setting (silence, vibration, or bell), time (dawn, morning, afternoon, or evening), and week or weekend. 4.2. Time-window We define time-window as a 1× N matrix, which divides a day into 288 contexts. Processed context factors generate a time-window every 5 minutes, and it is stored temporarily until the next processed factor occurs. Each time-window is given a unique ID by its chronological order. Table 1 shows some time-window samples. Each row signifies a context, which is composed of n number of context factors. Timewindows are used to train the user model as well as to detect context change by comparing the distance between target and incoming. 4.3 Context change detection We detect a context transition by calculating L2-norm Euclidean distance, distance(T,C), between target timewindow and incoming time-window. To start the C-C-D process, first, we have to set a negative event on smartNoti, which becomes a trigger to start this process. For example, we set a missed call and a rejected call as negative events in the callback reminder application. C-C-D only starts its process when these negative events occur, and stops when the message pops up. We set the default threshold as 2 . On a discrete variable basis, distance 2 means that there are two different discrete factors compared to the target time-window. Hence, we consider that this 2 signifies that the user context has been changed. From this default value, we dynamically adjust the default threshold until 3 . For instance, if the user calls back before receiving a notification, we regard it as early transition detection, distance(T,C) > θ n , and increase the threshold by formula (3) where α = 0.05 is the change rate. On the other hand, if the user does not recognize the reminder notification for a Table. 1 Time-window samples. ID 1 2 3 4 5
Activity Level 0 1.3 1.8 3,7 ….
Steps
Apps
Media
Setting
Time
0.049 0.16 0.78 0.062 ….
0 0.049 0.18 0 ….
False True False False ….
vibration silence bell vibration ….
morning morning afternoon afternoon ….
Week /ends week week week week ….
while, we regard it as late detection, distance(T,C) < θ n , and decrease the threshold by (3) where α = 0.05 is the change rate. (3)
θ n+1 = θ n + α (distance(T,C) - θ n ) 5. INCREMENTAL NAÏVE BAYES MODELING, CLASSIFICATION, AND NOTIFICATION
In this section, we design an incremental user model using the Naïve Bayesian (NB) approach as a baseline to identify whether the statistical model can be utilized in predicting user events. As NB only needs to update a few entries in its probability table, it consumes a much lower cost than a non-incremental approach, which has to rebuild a new classifier to include new training data. The real-time processing system running on a mobile device needs to consider not only computational cost, but also its limited memory space. We incrementally train the user model whenever events occur, and keep only a small amount of information in order to calculate posterior probability. The total count of discrete attributes is accumulated, and the online algorithm [5,6] helps to incrementally train continuous attributes by maintaining only three components; average, standard deviation, and m2. To classify an incoming context Ti as a suitable or unsuitable moment, we label each event with two hypotheses, H 0 (positive) and H1 (negative), and choose a maximum posteriori hypothesis by (5). P(H 0 | F1,..., Fn ) > or < P(H1 | F1,..., Fn ) (5) Posterior probability is shown in (6). n
P(H 0 | F1,..., Fn ) ≈ P(H 0 )⨿ P(Fi | H 0 )
(6)
i=1
By assuming that continuous attributes follow Gaussian distribution, posterior probability is calculated by (7). The probability of discrete attributes is (8). (F −µ )2
− i 2 1 (7) e 2σ σ 2π the number of fi in H 0 P(Fi = fi | H 0 ) = (8) total number of H 0 Lastly, the maximum posteriori hypothesis is calculated by log-likelihood ratios (9). If the context is closer to positive hypothesis H 0 , as in the case of incoming or outgoing calls, we immediately pop up a notification message, but if it is more close to H1 , we skip the context, and wait for another one. P(H 0 | F1,..., Fn ) P(H 0 ) n P(Fi | H 0 ) log = log + ∑ log (9) P(H1 | F1,..., Fn ) P(H1 ) i=1 P(Fi | H1 )
P(Fi = fi | H 0 ) =
Reinforcement study. To minimize the period of a cold start, and make the user model more robust, we try to utilize
all the following available opportunities in order to train the model. • Study positive model whenever positive event occurs. • Study negative model whenever negative event occurs. • Study negative model if user does not notice the notification message. • Study negative model if user does not react after checking message. 6. EXPERIMENTAL VALIDATION: INTELLIGENT CALLBACK REMINDER To evaluate smartNoti, we implemented the callback reminder application on an Android. Goal. To notify the user of callback reminder messages in the case of missed/rejected calls at the best suitable moment. Measuring parameter. Context factors (physical activity, step count, the amount of unique application usage, media usage, time, week/weekend), event factors (incoming call, outgoing call, missed call, rejected call), target factors (missed call, rejected call). Building model. Whenever event factors occur, the timewindow at the incoming, outgoing call moment is labeled as positive class, and the time-window at the missed, rejected call moment is labeled as negative class. Estimating state. To detect a context transition moment, the system compares the time-window at the missed/rejected call moment to the incoming time-window. Controlling the system. The model predicts whether the transition moment is a recognizable and available time, and then sends the reminder message. As a longitudinal study, we carried out user tests for about 2 months with 6 people who installed the application. To demonstrate the suitability of the predicted moment, we analyzed whether the issued notification was immediately recognized and regarded at a timely moment. 6.1 Implementation smartNoti is optimized for Android 4.4. The prototype has been tested on Samsung Galaxy S4, S5, LG G pro2, G3, Motorola Moto x (Gen 1). The callback reminder application has been implemented on smartNoti. The system is always running in the background as well as in the sleep mode. Test users must always be connected to the network as well as have the location turned on in Setting. We use call events (i.e. incoming, outgoing, rejection, miss) as event factors. The negative event is regarded as a trigger to start tracking the notification moment. We set a maximum tracking time, 1 hour, to prevent unlimited moment tracking. In addition, we release the tracking process when the user calls back missed or rejected numbers before receiving the alert. All processes are automatically processed as soon as the application is installed. A notification will pop up if smartNoti does not find the moment during the maximum
hour, but a reminder message will pop up if it finds a recognizable and available moment, and then requests the user feedback. The notification method is similar to SMS. 6.2 Evaluation method We evaluate the applicability of our approach in terms of the prediction of personalized user models for determining the recognizable and available moments. Also, we evaluate the user’s satisfaction regarding notification time. Prediction test. This tests the efficiency of the NB model. Based on the user model, we predict incoming events (negative or positive) and then compare the results to the real events. For example, when a missed call comes, we catch the context factors first, pass them through the user model to predict the real event, and check whether the predicted result is the same as the real event. User feedback test. To test the timeliness of the predicted moment, when the reminder notification pops up, we retrieve user feedback with a simple question “Is this a good time to call back?” If the user clicks on the “OK” button, we count the moment as TRUE. If the user clicks on “NO” or does not click any button immediately after the notification, we count the moment as FALSE. 6.3 Experimental results and discussion Table 2 shows the average accuracy of the prediction test and the user feedback test. The values in the parentheses indicate the rate of maximum-time notification. The boxplots in Figure 4 demonstrate the distribution of prediction accuracy among 6 users. To show the progress of model robustness, we drew the change of prediction accuracy in Figure 5. Each dot in Figure 5 indicates the average accuracy value for 5 days. Model effectiveness. The model predicts distinct positive or negative moments. A closer look at Table 2 reveals that although the prediction accuracy for subject 3 and subject 4 is relatively low, the feedback accuracy is high. This means that the Moment Predictor cannot classify several ambiguous cases (noise). For example, subject 3 missed a call in one time-window and called someone else in the same time-window. These vaguely trained contexts obstruct the exact moment classification. Also, the wide range of the distribution in Figure 4 and sudden accuracy decline of subject 1,3 in Figure 5 are due to noises in context formulation. However, the high feedback accuracy Table. 2 Average accuracy of prediction test and feedback test.
1 2 3 4 5 6
Precision 0.931 0.907 0.831 0.917 0.816 0.826
Recall 0.881 1 0.847 0.841 0.981 0.905
F-measure 0.905 0.951 0.839 0.878 0.891 0.863
Accuracy 0.832 0.907 0.735 0.785 0.804 0.778
Feedback 0.85(0.2) 0.9(0.2) 0.91(0.3) 0.83(0.2) 0.67(0.2) 1.0 (0.5)
Fig. 4. Distribution of prediction accuracy and feedback accuracy.
and demonstrated the potential of using an incremental learning algorithm. To evaluate the moment reasonability and efficiency of the personalized data model, we implemented a callback reminder on the smartNoti framework. The application was evaluated using two methods, a model prediction test and a user feedback test. In our future work, we will investigate the influences of other context factors such as location factors, logical factors, and optimize the proposed user model by applying these additional features and comparing the Naïve Bayes algorithm to other incremental classification algorithms. 8. REFERENCES
Fig. 5. Prediction accuracy over time
demonstrates that the model distinguishes the recognizable and available moments well. Prediction accuracy of the user model has gradually improved. Although Figure 5 shows a cold start during the first 10~15 days, the prediction accuracy of subjects increased over time. Even for subject 2 and 3, who used the system for only a short period of time, robustness progress is visible. User satisfaction. The user feedback test demonstrates how the detected moments are recognizable and available. Among all test subjects, only subject 5 shows some dissatisfaction with the recommended moments. We found that the low accuracy of this subject was because of the small amount of feedback data set. As discussed in Section 6.3, the efficiency of the NB model in smartNoti needs to be evaluated based on comprehensive results. To check the moment availability and recognizability, we also need to trust the results of the feedback test. In reviewing all of the results from the analysis in this section, we see that the suitable notification moments can be determined by smartNoti. 7. CONCLUSION AND FUTURE WORKS Finding a suitable moment is an important problem of the notification system. To completely convey need-to-know information, a moment should be immediately recognizable and available. However, most research on notification systems concentrates on either the recognizable moments or the message themselves. To solve this limitation, we proposed smartNoti, a mobile framework, which integrates, processes, trains, and stores personalized latent features, and detects context transition that might be the recognizable moments and predicts the moment availability. As part of identifying the effectiveness of the statistical model in moment prediction, we proposed an incremental NB model
[1] Mark Lycett, “Datafication: making sense of (big) data in a complex world,” European Journal of Information Systems, vol. 22, no. 4, pp. 381–386, July 2013. [2] Ramesh Jain, “Toward social life networks,” Computer, no. 11, pp. 86–88, 2014 [3] Abowd, G. D., Dey, A. K., Brown, P. J., Davies, N., Smith, M., & Steggles, P. (1999, January). Towards a better understanding of context and context-awareness. In Handheld and ubiquitous computing (pp. 304-307). [4] J. A. Nelder, “The Fitting of a Generalization of the Logistic Curve,” Biometrics, vol. 17, no. 1, pp. 89–110, Mar. 1961. [5] Tony F. Chan, Gene H. Golub, and Randall J. LeVeque, “Algorithms for Computing the Sample Variance: Analysis and Recommendations,” The American Statistician, vol. 37, no. 3, pp. 242–247, Aug. 1983. [6] Robert F. Ling, “Comparison of Several Algorithms for Computing Sample Means and Variances,” Journal of the American Statistical Association, vol. 69, no. 348, pp. 859–866, Dec. 1974. [7] Piotr D. Adamczyk and Brian P. Bailey, “If Not Now, when?: The Effects of Interruption at Different Moments Within Task Execution,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, New York, NY, USA, 2004, CHI ’04, pp. 271– 278, ACM. [8] Brian P. Bailey and Shamsi T. Iqbal, “Understanding Changes in Mental Workload During Execution of Goal directed Tasks and Its Application for Interruption Management,” ACM Trans. Computer-Human Interaction, vol. 14, no. 4, pp. 21:1–21:28, Jan. 2008. [9] Joyce Ho and Stephen S. Intille, “Using Context-aware Computing to Reduce the Perceived Burden of Interruptions from Mobile Devices,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, New York, NY, USA, 2005, CHI ’05, pp. 909– 918, ACM. [10] Nicky Kern, Bernt Schiele, "Context-Aware Notification for Wearable Computing. " 16th International Symposium on Wearable Computers, pp. 223-223. IEEE Computer Society, 2003.
