Tutorial Tutorial Goals

7 downloads 20801 Views 3MB Size Report
Particular attentiont to adaptation in Web applications ... It varies more with mobile devices than desktop ones ... Avoid overloading the user interface with many.
Adaptation and Continuity i Multiin Multi M lti-Device D i E Environments i t Fabio Paternò HIIS Laboratory ISTI-C.N.R. Pisa, Italy http://giove.isti.cnr.it/

Tutorial Goals 











How to obtain interfaces able to adapt to multiple device types while preserving usability? State of art in terms of approaches, approaches design criteria, criteria tools Particular attentiont to adaptation in Web applications and use of model-based techniques Understanding the space of the possible solutions in pp y them and think about new order to better apply solutions Consider how to address the device adaptation issue both at design time and at run-time Discussion on how adaptation and continuity can be 2 supported in migratory user interfaces

1

Structure         

Introduction, Basic Concepts, Issues Usability and Task in Multi Multi-device device environments Authoring Multi-Device Interfaces Model-based Support for Multi-device Interfaces Tools for Desktop-to-Mobile Adaptation Adaptation Customization Migratory Interfaces Partial / Trans-modal Migration Conclusions and Discussion

Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

3

Why Adaptation to the Context of Use User: • Preferences

Environment:

• Knowledge

•Position

• Goals

• Light

• Background

• Noise, …

•…

• Social aspects

Device:

•Privacy

• Screen resolution

•Collaboration

• Connectivity

•…….

• Browser •… 4

2

Display Size and Resolution     

Personal Computer (PC) usually varies between 800x600 and 1920x1200 pixels, Mobile devices usually between 240x240 and 960x640 pixels (Iphone 4) Simple phones have even lower resolutions It varies more with mobile devices than desktop ones The Moore Law continuously changes these numbers! b !

5

Mobile Interaction Techniques Variation http://www.w3.org/TR/mwabp http://www.w3.org/TR/ mwabp/#bp /#bp--presentation presentation--interaction



Focus Based: The browser focus moves through elements 





Pointer Based: Key-based navigation controls a pointer that can cover any part of the screen 







The current focus of the page is easily determined because the focus element will be highlighted; Focus area can move from one selectable element to another (e.g. (e g from link to link) even when widely spaced

Selectable elements that are associated with each other need to be close as moving the pointer can be slow; Selectable elements need to be large enough to be easily selected -- since the pointer often moves in steps of between 5 - 10 pixels; Selectable elements should have rollovers to make it clear when the pointer has entered their active area.

Touch Based: Events are related directly to a finger or stylus touch position on the screen.  



Selectable elements may be widely spaced since the user can select them directly; Selectable elements must be large enough to be easily selected (e.g. list items should have a height of at least 30px); No elements are in focus until they are selected so extra information cannot be passed to the user (e.g. rollovers will not work).

Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

6

3

Usability in Mobile Interaction         

Minimize text input Exploit the elements of the mobile device, such y as accesskeys Consistency between platforms Prevent user error The purpose of the interface elements should be clear Avoid overloading the user interface with many elements Limit the need for scrolling Short time access Access to small pieces of information

Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

7

Supporting Tasks in Multi Multi-platform Environments  









Platform definition Same task on multiple p p platforms in the same manner Same task on multiple platforms but with different user interface elements Same main task, with different levels of subtasks Dependencies among tasks performed on different platforms Tasks meaningful only on some platform types Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

8

4

Same task on multiple platforms with different user interfaces

Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

9

Task Meaningful only on Some Platforms

Search for flights Promotions Reservations Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

10

5

Authoring MultiMulti-device Interfaces 



 

Platform-specific Platform specific authoring (e.g. (e g Amazon) Multiple-device authoring (e.g. with CSS) Single authoring Automatic re-authoring

Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

11

Multiple Device Authoring Damask (James Lin PhD – CHI’08)

•Sketches •Layers •Patterns

Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

12

6

Automatic Re Re--Authoring  



Scaling, such as Safari on IPhone T Transducing, d i t translates l t elements l t iinto t other formats, and compresses and converts images to match device characteristics, such as Mowser or Skweezer, T Transforming f i goes further f th tto modify dif b both th contents and structures originally designed for desktop systems to make them suitable to display on small screens

Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

13

Transforming (examples) 





Single column, (for example Opera SSR)

eliminates scrolling in one dimension, it greatlyy increases the amount of scrolling g g in the other dimension. Fisheye (for example Fishnet) is a fisheye Web browser that shows a focus region at a readable scale while spatially compressing page content outside the focus region Overview + detail splits a Web page into multiple l i l sections i and d provides id an overview i page with links to these sections.

Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

14

7

Narrow Solution (e.g. Opera SSR)

………. A lot of vertical scrolling !!!!

15

Narrow Solution 





 

The order for the content follows that of the markup file starting with the top The images are scaled to the size of the screen The text is always visible and the content is compacted without blank spaces Content that requires space such as maps and tables can become unreadable Sometimes it is difficult to understand that the page has changed because the initial part is the same 16

8

Requirements for the Nokia Solution (CHI 2006, Roto et al.) 



 

Remove the need of horizontal scrolling to read text Provide enough contextual information to give an idea of the page structure and the current location in it Don’t destroy the original page layout Don’t introduce modal interaction

Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

17

MiniMap Method (CHI 2006, Roto et al.) Overview + Detail

Nokia S60 phones

Original

Compacted

Mini-Map

Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

18

9

Application of Information Visualization Techniques Fisheye calendar allows complex tasks to be completed more quickly

Focus + Context In DateLens [Bederson et al 02, 04] 19

Model--based approaches Model 

 







Allow designers and developers to concentrate on main semantic aspects L Languages th t representt such that h aspects t Avoid need to learn and manage many implementation languages Linking semantic information and implemementation p elements Interoperability through many possible implementation languages Facilitate support of assistive technology

Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

20

10

Abstraction Levels in Interactive Systems 

Task and object – Activity oriented 



Abstract Interface – Platform Independent 



– Single selection object with high cardinality

Concrete Interface –Platform Dependent 



– I want to select a work of art

– List Interaction object with X elements

Implementation 

– List object in Java or XHTML or ....

Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

21

The ConcurTaskTrees Notation for Task Models Hierarchical structure

Temporal relations

Task Allocation

Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

22

11

Support for Applications based on Web Services (http://giove.isti.cnr.it/tools/Mariae) Task  Model

Web  Services

TM Language

Desktop AUI Specification

PDA AUI  Specification

Multitouch  Phone AUI  Specification

Desktop CUI  Specification

PDA CUI  Specification

Multitouch  Phone CUI  Specification

MARIA

AUI  Language

Annotations

Specifications

M‐touch     Phone         CUI  Language

PDA     CUI  Language

Desktop p CUI  Language

Languages

Reverse Engineering Automatic Model Creation !

Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

24

12

Adaptation in Authoring Environments (SUPPLE, Gajos and Weld) Weld) 





Takes a functional specification of the interface, the devicespecific constraints, a typical usage trace, and a cost function. The cost function is based on user preferences and expected speed of operation SUPPLE’s optimization algorithm, finds the user interface which minimizes the cost function while also satisfying all device constraints.

Adaptation and Continuity in Multi-Device Environments

25

Architectural Solutions for Automatic Web Adaptation Application server-side adaptation Request/Device capability

Server

Client Adapted Content

Proxy-side adaptation Request

Server

Proxy

Request/ Device capability

Client

Adapted content

Content

Client-side adaptation Request

Server

Client Content

Adapted content

Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

26

13

Dynamic User Interface Adaptation   

Use of Logical Description Languages Existing Web desktop Applications Automatic user interface generation

Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

Parametric Bidimensional Cost--based Semantic Redesign Cost Font adjustments according user preferences

Images Interactor replacement Long texts

Grouping Relation Data tables Layout tables

Paternò, Zichitella, HCSE 2010

14

Adaptation Solutions http://mowser.com

•Uses predefined style sheets •No support for tables or long texts

http://www.skweezer.com

•Limited reduction of image dimensions •Aims to reduce horizontal scrolling

Customizing Adaptation

Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

30

15

Customizing Adaptation

31

Browsing Large Table in Small Screens

Ohnishi and Tajima, UIST’08

Ease comparison of cells far from each other Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

32

16

Page Summarization 



The Abstraction-based approach uses sentence manipulation techniques like reduction, reduction compression and reformulation. The Extraction-based approach assigns scores to sentences in order to select those which better represents the whole text 

Feature based (e.g. term frequency, sentence position,, attributes…); p ); machine learning, g, graph g p based techniques

Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

33

Micro--level Summarization Micro

(PowerBrowser, PowerBrowser, Buyukkokten et al.) 



 



The importance of a keyword depends on the frequency it occurs in a text and in a larger collection A word within a given text is considered most important if it occurs frequently within the text, but infrequently in the larger collection The significance factor of a sentence is derived from an analysis of its constituent words The e se sentences te ces in which c tthe eg greatest eatest number u be o of frequently occurring distinct words are found in closest proximity to each other are probably important MEAD is a public multi-document summarization system, which provides more flexible support in this area (see http://www.summarization.com/mead/)

17

Adaptation in MultiMulti-User and Multi--Device environments Multi 

WebSplitter (Han et al., CSCW’00)- collaborative Web browsing by creating personalized partial views of the same Web page depending on the user and the device. Developers have to specify the Web content in XML and define a policy file indicating what content tags should be shown for each device and user.

Concepts and Tools for Multi-Device User Interfaces

35

How People Use Multiple Devices (Dearman & Pierce ‘08) 







A recent study in US of 27 people from academic and industrial research revealed that on average they employ more than five computing devices Associating a user’s activities with a particular device is problematic for multiple device users because many activities span multiple devices Device use varies by user and circumstance; users assign different roles to devices both by choice and by constraint. Users employ a variety of techniques for accessing information across devices but participants reported managing information across their devices as the most challenging aspect of using multiple devices. 36

18

Flexible Access in MultiMultiDevice environments 





Moving objects: across interactive d i devices th through h pick-and-drop i k dd Distributed user interfaces: application logic receiving input from multiple devices Migratory g y user interfaces: change g device, interface migration with state preservation

Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

37

Pick--and Pick and--drop

Jun Rekimoto, UIST’97 / CHI’98

A Direct-manipulation Technique for Multiple-Computer Environments

38

19

PUC (Personal Universal Controller) [CMU, Nichols, Myers] Personal Mobile Device

Automatically G Generated t d Interface

Abstract Specification

Control State Feedback

39

Why Migratory Interfaces  







Our life is becoming a multi-device experience One of the main source of frustration is that we need to restart for each device change Need for continuous access to interactive services across various devices Migratory user interfaces can transfer among different devices ((from ‘source’ devices to ‘target’ devices), so as to allow the users to continue their tasks Application domains such as shopping, bids for auction on line, games, making reservations

Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

40

20

Migration Phases       

Device Discovery: Devices notify their presence When to Migrate: Migration Trigger (User or System or Mixed initiative) Where to Migrate: Identification of Target and its resources What to Migrate: User interface and/or Application logic How to Migrate: Adaptation (depending on the type of target) State Persistence: Source state extraction and association to target version Activation in the target device: Upload the adapted version at the point in which the user left 41 off on the source device

Usability in Migration 

Relevant aspects to continuity 

Time  



Adaptation process 



When adaptation makes unclear how to continue the task

Predictability:   



Time required d by b the h migration process Time from the last interaction in the source device

The target device Which part of the user interface migrate Where the result of an interaction will be presented

Learning: to get familiar with the migration

process

42

21

What is the state to preserve       

User Input Focus Cookies Session History Bookmarks J JavaScripts S i t Task: SelectMenu value: Vegetable

Your Selected Menu is Vegetable

43

Web Session Migration Using Dynamic 2D Barcodes A. Alapetite, PUC 14(1): 45-52 (2010)

Append the ID of the current session (token) to the current URL query string

When the Web server receives a request from a new device (the mobile phone) with a known existing session ID, the server will provide the same content as the one served to the previous device (the desktop). Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

44

22

Approaches to Migration 







Bharat and Cardelli (UIST’95) - the migration of entire applications Chung and Ch d Dewan D (UIST’96) - when h migration i ti is i triggered the environment starts a fresh copy of the application process in the target system, and replays the saved sequence of input events to the copy. This solution does not support interface adaptation. Kozuch and Satyanarayanan (2002) - solution for migration based on the encapsulation of all volatile execution state of a virtual machine (migration of applications among desktop or laptop systems) Melchior, Grolaux, Vanderdonckt (EICS’09) - solution for distributed user interfaces with extension of Tcl/Tk

Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

45

Architectural Framework Device 1 Application • User Interface • Application Logic • Connection to OPEN Client

Migration Platform Mi Migration ti Framework F k •Server Migration Services Orchestration

OPEN Client • Runs R iin b background k d • Connects to OPEN platform framework

Migration g Services User Interface migration Application logic migration State persistence Context management Trigger management

46

23

Device discovery and selection Target Device Selection •same environment •device availability •same user (or shared) •interaction features

Example representation of the migration environment Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

47

Device Selection

Ghiani, PhD

Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

48

24

Example Architecture Migration Device Discovery (1)

Migration Trigger (6)

Migration Client Request Page (2)

Desktop Application

Cookies Provide Annotated Page (5)

Reverse Semantic Redesign State Mapper Generator

Proxy Server

Upload (8)

Trasmission of DOM + current state through callback AJAX (7)

Provide Page (4)

PDA

Request Page (3)

Application Server

49

Migrating between mobile and Digital Tv

Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

50

25

Example Migration (Video) (shopping scenario)

Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

51

Trans--Modal Migration Trans



Vocabularyy Information organization Different concrete objects



Not a simple one to one mapping

 

 



Sender

Subject

Date &Time

Size

Luois lestat

Last minute offer

May 25 9:23

1k

George Clow

Re: Cameleon meeting

June 3 11:02

3k

Luois Lestat

Learn about Java

June 5 12:20

2k

June 7 10:00

1k

Mary Rex Re: Cameleon meeting

Task not Supported Task implemented with different concrete objects Adaptation of runtime data

52

26

Differences in task support implementation (grouping sound) Please say your name. Please say y your y surname. Please say the date of reservation. What type of menu do you prefer: fish; meat; vegetable (grouping sound) Your name is …. Your surname is …. You have booked a table on … and You prefer… (grouping sound) If you want to cancel the reservation , say delete; If you want to confirm the reservation, say ok (grouping sound). Grouping Operator

Object that Support the task “Provide date”

53

Adaptation of runtime data Your name is Louis Louis. You have booked a table on 3 July. July (grouping sound) Please say your name. Please say your surname. Name: Louis Date: 3 July

Please say the date of reservation. What type of menu do you prefer: fish; meat; vegetable (grouping sound) Your name is …. Your surname is …. You have booked a table on … and You prefer… (grouping sound) If you want to cancel the reservation , say delete; If you want to confirm the reservation, say ok (grouping sound). 54

27

Application Logic Reconfiguration

Clausthal University in OPEN Project

Partial Migration (Mobile HCI 2010 paper)

56

28

Partial Migration

Partial Migration with Interface Structure Representation

29

Partial Migration with Direct Selection

59

Partial Migration with Direct Selection (video)

Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

60

30

Research Agenda   





 

Automatic Desktop-to-Vocal Adaptation Migration g involving g multiple p users Integration of wider set of interaction modalities in migratory environments Migration from multiple devices to multiple devices Authoring Environments for Distributed User I t f Interfaces Privacy and Security of Migratory User Interfaces Applying Machine Learning Techniques to User Interface Adaptation Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

61

More Info at   

EU OPEN Project http://www.ict-open.eu EU SERENOA Project http://www.serenoa-fp7.eu/ htt // f 7 / W3C group on model-based interfaces http://www.w3.org/2005/Incubator/model-based-ui/charter/



EU Artemis SMARCOS Project

http://www.smarcos-

project.eu/ 

Forthcoming F th i Spinger S i Book B k on Migratory Mi t Interactive Applications in Ubiquitous Environments Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

62

31

Don’t Forget EICS 2011 !

Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it

63

32