Particular attentiont to adaptation in Web applications ... It varies more with mobile devices than desktop ones ... Avoid overloading the user interface with many.
Adaptation and Continuity i Multiin Multi M lti-Device D i E Environments i t Fabio Paternò HIIS Laboratory ISTI-C.N.R. Pisa, Italy http://giove.isti.cnr.it/
Tutorial Goals
How to obtain interfaces able to adapt to multiple device types while preserving usability? State of art in terms of approaches, approaches design criteria, criteria tools Particular attentiont to adaptation in Web applications and use of model-based techniques Understanding the space of the possible solutions in pp y them and think about new order to better apply solutions Consider how to address the device adaptation issue both at design time and at run-time Discussion on how adaptation and continuity can be 2 supported in migratory user interfaces
1
Structure
Introduction, Basic Concepts, Issues Usability and Task in Multi Multi-device device environments Authoring Multi-Device Interfaces Model-based Support for Multi-device Interfaces Tools for Desktop-to-Mobile Adaptation Adaptation Customization Migratory Interfaces Partial / Trans-modal Migration Conclusions and Discussion
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
3
Why Adaptation to the Context of Use User: • Preferences
Environment:
• Knowledge
•Position
• Goals
• Light
• Background
• Noise, …
•…
• Social aspects
Device:
•Privacy
• Screen resolution
•Collaboration
• Connectivity
•…….
• Browser •… 4
2
Display Size and Resolution
Personal Computer (PC) usually varies between 800x600 and 1920x1200 pixels, Mobile devices usually between 240x240 and 960x640 pixels (Iphone 4) Simple phones have even lower resolutions It varies more with mobile devices than desktop ones The Moore Law continuously changes these numbers! b !
5
Mobile Interaction Techniques Variation http://www.w3.org/TR/mwabp http://www.w3.org/TR/ mwabp/#bp /#bp--presentation presentation--interaction
Focus Based: The browser focus moves through elements
Pointer Based: Key-based navigation controls a pointer that can cover any part of the screen
The current focus of the page is easily determined because the focus element will be highlighted; Focus area can move from one selectable element to another (e.g. (e g from link to link) even when widely spaced
Selectable elements that are associated with each other need to be close as moving the pointer can be slow; Selectable elements need to be large enough to be easily selected -- since the pointer often moves in steps of between 5 - 10 pixels; Selectable elements should have rollovers to make it clear when the pointer has entered their active area.
Touch Based: Events are related directly to a finger or stylus touch position on the screen.
Selectable elements may be widely spaced since the user can select them directly; Selectable elements must be large enough to be easily selected (e.g. list items should have a height of at least 30px); No elements are in focus until they are selected so extra information cannot be passed to the user (e.g. rollovers will not work).
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
6
3
Usability in Mobile Interaction
Minimize text input Exploit the elements of the mobile device, such y as accesskeys Consistency between platforms Prevent user error The purpose of the interface elements should be clear Avoid overloading the user interface with many elements Limit the need for scrolling Short time access Access to small pieces of information
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
7
Supporting Tasks in Multi Multi-platform Environments
Platform definition Same task on multiple p p platforms in the same manner Same task on multiple platforms but with different user interface elements Same main task, with different levels of subtasks Dependencies among tasks performed on different platforms Tasks meaningful only on some platform types Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
8
4
Same task on multiple platforms with different user interfaces
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
9
Task Meaningful only on Some Platforms
Search for flights Promotions Reservations Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
10
5
Authoring MultiMulti-device Interfaces
Platform-specific Platform specific authoring (e.g. (e g Amazon) Multiple-device authoring (e.g. with CSS) Single authoring Automatic re-authoring
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
11
Multiple Device Authoring Damask (James Lin PhD – CHI’08)
•Sketches •Layers •Patterns
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
12
6
Automatic Re Re--Authoring
Scaling, such as Safari on IPhone T Transducing, d i t translates l t elements l t iinto t other formats, and compresses and converts images to match device characteristics, such as Mowser or Skweezer, T Transforming f i goes further f th tto modify dif b both th contents and structures originally designed for desktop systems to make them suitable to display on small screens
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
13
Transforming (examples)
Single column, (for example Opera SSR)
eliminates scrolling in one dimension, it greatlyy increases the amount of scrolling g g in the other dimension. Fisheye (for example Fishnet) is a fisheye Web browser that shows a focus region at a readable scale while spatially compressing page content outside the focus region Overview + detail splits a Web page into multiple l i l sections i and d provides id an overview i page with links to these sections.
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
14
7
Narrow Solution (e.g. Opera SSR)
………. A lot of vertical scrolling !!!!
15
Narrow Solution
The order for the content follows that of the markup file starting with the top The images are scaled to the size of the screen The text is always visible and the content is compacted without blank spaces Content that requires space such as maps and tables can become unreadable Sometimes it is difficult to understand that the page has changed because the initial part is the same 16
8
Requirements for the Nokia Solution (CHI 2006, Roto et al.)
Remove the need of horizontal scrolling to read text Provide enough contextual information to give an idea of the page structure and the current location in it Don’t destroy the original page layout Don’t introduce modal interaction
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
17
MiniMap Method (CHI 2006, Roto et al.) Overview + Detail
Nokia S60 phones
Original
Compacted
Mini-Map
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
18
9
Application of Information Visualization Techniques Fisheye calendar allows complex tasks to be completed more quickly
Focus + Context In DateLens [Bederson et al 02, 04] 19
Model--based approaches Model
Allow designers and developers to concentrate on main semantic aspects L Languages th t representt such that h aspects t Avoid need to learn and manage many implementation languages Linking semantic information and implemementation p elements Interoperability through many possible implementation languages Facilitate support of assistive technology
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
20
10
Abstraction Levels in Interactive Systems
Task and object – Activity oriented
Abstract Interface – Platform Independent
– Single selection object with high cardinality
Concrete Interface –Platform Dependent
– I want to select a work of art
– List Interaction object with X elements
Implementation
– List object in Java or XHTML or ....
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
21
The ConcurTaskTrees Notation for Task Models Hierarchical structure
Temporal relations
Task Allocation
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
22
11
Support for Applications based on Web Services (http://giove.isti.cnr.it/tools/Mariae) Task Model
Web Services
TM Language
Desktop AUI Specification
PDA AUI Specification
Multitouch Phone AUI Specification
Desktop CUI Specification
PDA CUI Specification
Multitouch Phone CUI Specification
MARIA
AUI Language
Annotations
Specifications
M‐touch Phone CUI Language
PDA CUI Language
Desktop p CUI Language
Languages
Reverse Engineering Automatic Model Creation !
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
24
12
Adaptation in Authoring Environments (SUPPLE, Gajos and Weld) Weld)
Takes a functional specification of the interface, the devicespecific constraints, a typical usage trace, and a cost function. The cost function is based on user preferences and expected speed of operation SUPPLE’s optimization algorithm, finds the user interface which minimizes the cost function while also satisfying all device constraints.
Adaptation and Continuity in Multi-Device Environments
25
Architectural Solutions for Automatic Web Adaptation Application server-side adaptation Request/Device capability
Server
Client Adapted Content
Proxy-side adaptation Request
Server
Proxy
Request/ Device capability
Client
Adapted content
Content
Client-side adaptation Request
Server
Client Content
Adapted content
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
26
13
Dynamic User Interface Adaptation
Use of Logical Description Languages Existing Web desktop Applications Automatic user interface generation
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
Parametric Bidimensional Cost--based Semantic Redesign Cost Font adjustments according user preferences
Images Interactor replacement Long texts
Grouping Relation Data tables Layout tables
Paternò, Zichitella, HCSE 2010
14
Adaptation Solutions http://mowser.com
•Uses predefined style sheets •No support for tables or long texts
http://www.skweezer.com
•Limited reduction of image dimensions •Aims to reduce horizontal scrolling
Customizing Adaptation
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
30
15
Customizing Adaptation
31
Browsing Large Table in Small Screens
Ohnishi and Tajima, UIST’08
Ease comparison of cells far from each other Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
32
16
Page Summarization
The Abstraction-based approach uses sentence manipulation techniques like reduction, reduction compression and reformulation. The Extraction-based approach assigns scores to sentences in order to select those which better represents the whole text
Feature based (e.g. term frequency, sentence position,, attributes…); p ); machine learning, g, graph g p based techniques
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
33
Micro--level Summarization Micro
(PowerBrowser, PowerBrowser, Buyukkokten et al.)
The importance of a keyword depends on the frequency it occurs in a text and in a larger collection A word within a given text is considered most important if it occurs frequently within the text, but infrequently in the larger collection The significance factor of a sentence is derived from an analysis of its constituent words The e se sentences te ces in which c tthe eg greatest eatest number u be o of frequently occurring distinct words are found in closest proximity to each other are probably important MEAD is a public multi-document summarization system, which provides more flexible support in this area (see http://www.summarization.com/mead/)
17
Adaptation in MultiMulti-User and Multi--Device environments Multi
WebSplitter (Han et al., CSCW’00)- collaborative Web browsing by creating personalized partial views of the same Web page depending on the user and the device. Developers have to specify the Web content in XML and define a policy file indicating what content tags should be shown for each device and user.
Concepts and Tools for Multi-Device User Interfaces
35
How People Use Multiple Devices (Dearman & Pierce ‘08)
A recent study in US of 27 people from academic and industrial research revealed that on average they employ more than five computing devices Associating a user’s activities with a particular device is problematic for multiple device users because many activities span multiple devices Device use varies by user and circumstance; users assign different roles to devices both by choice and by constraint. Users employ a variety of techniques for accessing information across devices but participants reported managing information across their devices as the most challenging aspect of using multiple devices. 36
18
Flexible Access in MultiMultiDevice environments
Moving objects: across interactive d i devices th through h pick-and-drop i k dd Distributed user interfaces: application logic receiving input from multiple devices Migratory g y user interfaces: change g device, interface migration with state preservation
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
37
Pick--and Pick and--drop
Jun Rekimoto, UIST’97 / CHI’98
A Direct-manipulation Technique for Multiple-Computer Environments
38
19
PUC (Personal Universal Controller) [CMU, Nichols, Myers] Personal Mobile Device
Automatically G Generated t d Interface
Abstract Specification
Control State Feedback
39
Why Migratory Interfaces
Our life is becoming a multi-device experience One of the main source of frustration is that we need to restart for each device change Need for continuous access to interactive services across various devices Migratory user interfaces can transfer among different devices ((from ‘source’ devices to ‘target’ devices), so as to allow the users to continue their tasks Application domains such as shopping, bids for auction on line, games, making reservations
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
40
20
Migration Phases
Device Discovery: Devices notify their presence When to Migrate: Migration Trigger (User or System or Mixed initiative) Where to Migrate: Identification of Target and its resources What to Migrate: User interface and/or Application logic How to Migrate: Adaptation (depending on the type of target) State Persistence: Source state extraction and association to target version Activation in the target device: Upload the adapted version at the point in which the user left 41 off on the source device
Usability in Migration
Relevant aspects to continuity
Time
Adaptation process
When adaptation makes unclear how to continue the task
Predictability:
Time required d by b the h migration process Time from the last interaction in the source device
The target device Which part of the user interface migrate Where the result of an interaction will be presented
Learning: to get familiar with the migration
process
42
21
What is the state to preserve
User Input Focus Cookies Session History Bookmarks J JavaScripts S i t Task: SelectMenu value: Vegetable
Your Selected Menu is Vegetable
43
Web Session Migration Using Dynamic 2D Barcodes A. Alapetite, PUC 14(1): 45-52 (2010)
Append the ID of the current session (token) to the current URL query string
When the Web server receives a request from a new device (the mobile phone) with a known existing session ID, the server will provide the same content as the one served to the previous device (the desktop). Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
44
22
Approaches to Migration
Bharat and Cardelli (UIST’95) - the migration of entire applications Chung and Ch d Dewan D (UIST’96) - when h migration i ti is i triggered the environment starts a fresh copy of the application process in the target system, and replays the saved sequence of input events to the copy. This solution does not support interface adaptation. Kozuch and Satyanarayanan (2002) - solution for migration based on the encapsulation of all volatile execution state of a virtual machine (migration of applications among desktop or laptop systems) Melchior, Grolaux, Vanderdonckt (EICS’09) - solution for distributed user interfaces with extension of Tcl/Tk
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
45
Architectural Framework Device 1 Application • User Interface • Application Logic • Connection to OPEN Client
Migration Platform Mi Migration ti Framework F k •Server Migration Services Orchestration
OPEN Client • Runs R iin b background k d • Connects to OPEN platform framework
Migration g Services User Interface migration Application logic migration State persistence Context management Trigger management
46
23
Device discovery and selection Target Device Selection •same environment •device availability •same user (or shared) •interaction features
Example representation of the migration environment Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
47
Device Selection
Ghiani, PhD
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
48
24
Example Architecture Migration Device Discovery (1)
Migration Trigger (6)
Migration Client Request Page (2)
Desktop Application
Cookies Provide Annotated Page (5)
Reverse Semantic Redesign State Mapper Generator
Proxy Server
Upload (8)
Trasmission of DOM + current state through callback AJAX (7)
Provide Page (4)
PDA
Request Page (3)
Application Server
49
Migrating between mobile and Digital Tv
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
50
25
Example Migration (Video) (shopping scenario)
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
51
Trans--Modal Migration Trans
Vocabularyy Information organization Different concrete objects
Not a simple one to one mapping
Sender
Subject
Date &Time
Size
Luois lestat
Last minute offer
May 25 9:23
1k
George Clow
Re: Cameleon meeting
June 3 11:02
3k
Luois Lestat
Learn about Java
June 5 12:20
2k
June 7 10:00
1k
Mary Rex Re: Cameleon meeting
Task not Supported Task implemented with different concrete objects Adaptation of runtime data
52
26
Differences in task support implementation (grouping sound) Please say your name. Please say y your y surname. Please say the date of reservation. What type of menu do you prefer: fish; meat; vegetable (grouping sound) Your name is …. Your surname is …. You have booked a table on … and You prefer… (grouping sound) If you want to cancel the reservation , say delete; If you want to confirm the reservation, say ok (grouping sound). Grouping Operator
Object that Support the task “Provide date”
53
Adaptation of runtime data Your name is Louis Louis. You have booked a table on 3 July. July (grouping sound) Please say your name. Please say your surname. Name: Louis Date: 3 July
Please say the date of reservation. What type of menu do you prefer: fish; meat; vegetable (grouping sound) Your name is …. Your surname is …. You have booked a table on … and You prefer… (grouping sound) If you want to cancel the reservation , say delete; If you want to confirm the reservation, say ok (grouping sound). 54
27
Application Logic Reconfiguration
Clausthal University in OPEN Project
Partial Migration (Mobile HCI 2010 paper)
56
28
Partial Migration
Partial Migration with Interface Structure Representation
29
Partial Migration with Direct Selection
59
Partial Migration with Direct Selection (video)
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
60
30
Research Agenda
Automatic Desktop-to-Vocal Adaptation Migration g involving g multiple p users Integration of wider set of interaction modalities in migratory environments Migration from multiple devices to multiple devices Authoring Environments for Distributed User I t f Interfaces Privacy and Security of Migratory User Interfaces Applying Machine Learning Techniques to User Interface Adaptation Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
61
More Info at
EU OPEN Project http://www.ict-open.eu EU SERENOA Project http://www.serenoa-fp7.eu/ htt // f 7 / W3C group on model-based interfaces http://www.w3.org/2005/Incubator/model-based-ui/charter/
EU Artemis SMARCOS Project
http://www.smarcos-
project.eu/
Forthcoming F th i Spinger S i Book B k on Migratory Mi t Interactive Applications in Ubiquitous Environments Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
62
31
Don’t Forget EICS 2011 !
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it
63
32