NotePals: Sharing and Synchronizing Handwritten Notes with Multimedia Documents
James A. Landay, Richard C. Davis, Victor Chen, Jonathan Huang, Rebecca B. Lee, Francis Li, James Lin, Charles B. Morrey III, and Ben Schleimer
Group for User Interface
Research
EECS Department
University of California
Berkeley, CA 94720-1776 USA
+1 510 643 7354
{landay,
rcdavis}@cs.berkeley.edu
ABSTRACT
NotePals is an ink-based, collaborative note taking application that runs on personal digital assistants (PDAs). Meeting participants write notes in their own handwriting on a PDA. These notes are shared with other participants by synchronizing later with a shared note repository that can be viewed using a desktop-based web browser. NotePals is distinguished by its lightweight process, interface, and hardware. This demonstration illustrates the design of two different NotePals clients and our web-based note browser.
Keywords
PDA, pen-based user interface, CSCW, informal user interfaces, gestures, digital ink, mobile computing
INTRODUCTION
NotePals is an ink-based, collaborative note taking application that runs on pen-based devices (e.g., PDAs). The system is distinguished by its support for lightweight collaboration at three levels: hardware, note taking process, and user interface. Our initial prototype runs on the 3Com PalmPilot, which weighs only 5.7 ounces (165 grams), easily fits in one’s palm, and sells for under $300 USD. The note taking process allows each participant in a meeting to take his or her own notes in free-form ink. NotePals stores these notes in a shared repository so that these notes can augment (or possibly replace) other meeting records.
The NotePals ink-based user interface uses a zoomed view that attempts to overcome problems associated with the small size of PDAs (see Figure 1a). This allows users to focus on taking notes quickly using their own handwriting without relying on error-prone handwriting recognizers or unfamiliar shorthands [1], such as Graffiti.
The drive to create NotePals came from the concern that people often leave meetings without a shared understanding of the important points that occurred. Assigning a scribe to record minutes is one solution, but it is onerous and can produce a biased record. Using computer-based meeting support tools is another solution, but existing tools require an expensive, fixed infrastructure that limits the locations where meetings can be held.
We felt that small, inexpensive PDAs might provide a better platform for meeting support tools. Meeting participants use NotePals on PDAs for taking notes during a meeting. Afterwards, the participants synchronize their PDAs with their own desktop machines, and their notes are sent to a shared repository stored on a server. The participants can then use a web browser on their desktop computer to view these merged notes. They can sort and filter the notes by time, project, author, date, and note type. Figure 2 illustrates a merged set of notes taken during the UIST ’97 conference.
NOTEPALS
While the Pilot’s size makes it easy to carry, it makes it very difficult to draw on. The Pilot’s writing surface is so tiny that user’s hands obstruct their view of the screen while drawing. In addition, the Pilot’s 160 x 160 pixel resolution makes it difficult to write small. This situation is not likely to improve soon, since the Pilot’s size is part of what makes it so popular.
PalmPilot User Interface
A NotePals "note" is a single screen "chunk" of text and other scribbles (see Figure 1). Drawing directly on the page of notes works well for sketches, but for text the focus window in the bottom portion of the screen is used. A small box (the "cursor") indicates the focus window’s current view in the page of shrunken notes at the top of the screen (see Figure 1a). Words written in the focus window will also appear above inside the cursor scaled down by a factor of 2½. This gives each page a total resolution of 400 x 273.
Figure 1. NotePals' user interface with (a) focus area active and (b) note attributes area active.
This design allows the user to fit more text on a page, and it keeps the user’s hand out of the way while writing. As the user writes, he can make a right to left gesture in the focus window to move the cursor forward. A down and then left gesture moves the cursor to the start of the next line. Users can also move the cursor by dragging it in the context area.
A page’s "stationery type" indicates what kind of information is in the note. "Note", the default, is the simplest type and is treated like a plain piece of paper with writing. Notes can be given more specific types that include additional attributes, such as "Action Items" which have due date and owner attributes (see Figure 1b).
Web-based Note Repository
The note repository is simply a web server that accepts uploaded notes from the desktop that the Pilot synchronizes with. The server is responsible for storing and sorting all the notes uploaded to it. The system currently stores the attributes of the notes in an SQL database, while the notes themselves are stored in invidividual files.
Notes are interleaved by time on a page by page basis, and can be browsed with a simple web interface that allows the user to filter and sort them (see Figure 2) . The user can filter the data by note attributes. Attributes and stationery types allow the user to form more complex queries such as, "Show all the action items Harvey took yesterday." Clicking on one of the attribute names will sort the notes by that attribute. While notes may be difficult to read on the Pilot, clicking on a thumbnail in the Note Browser displays the note at full size.
Figure 2. The Note Browser displaying notes from UIST '97.
USAGE EXPERIENCE
We have run a group note taking study [2] which concluded that shared notes work better when there is a "document" around which to structure the notes (e.g., an agenda, presentation slides, or typed minutes). This led to novel uses of NotePals that we had not previously considered.
NotePals with Electronic Conference Proceedings
A example of a novel use of NotePals was our experience with it at the CHI '98 conference. Six members of our group took over 300 "pages" of NotePals notes during the conference. We have merged the conference notes, organized by paper title, and placed them on "top" of the papers in the electronic conference proceedings (see Figure 3). We found these notes useful for our post-conference trip meeting to discuss what was interesting in this busy multi-track conference. The entire set of papers and notes can be accessed from http://guir.berkeley.edu/classes/notes/html/index2.html.
We can use electronic versions of both the conference program and proceedings to structure notes both before and after a conference. If the program can be uploaded to the web repository before the conference, the title of each talk can be downloaded to the PDAs of all group members. At the beginning of a new talk, the user can simply select the talk's title from a list and begin taking notes. After the conference, each group member's notes can be made available alongside the conference proceedings themselves (see Figure 3).
When browsing these proceedings, other group members can easily see their colleague's valuable ideas and opinions. The confusion that could result from having so many perspectives in one document is avoided by using the proceedings themselves as a unifying document. The proceedings also serve as context necessary for the group members to recall the subject of their notes.
Figure 3. A CHI '98 paper along with our notes taken during the conference presentation.
Figure 3 illustrates this situation with the notes taken during the CHI '98 conference. The notes taken during a specific talk are displayed in a window on top of the corresponding conference proceeding, obtained from the ACM Digital Library. The user can cycle through all the notes written by group members during the talk, toggle between the original handwriting and recognized ASCII text, and open additional windows to allow more than one note to be viewed at a time.
This application has been implemented as a Java applet, allowing it to be viewed by Java-enabled web browsers. The applet uses the Multivalent Document Architecture (MVD) [3]. MVD documents are composed of "layers" of related data and dynamically loaded "behaviors" that provide the functionality to manipulate the layers. For example, the conference proceeding applet is comprised of both a scanned image layer and a layer of extracted text generated by an OCR process. The selection behavior allows the user to perform selection actions on the scanned image while manipulating the corresponding text; i.e. performing a copy command will copy the text, not the image of the selected area. Similarly, each note is comprised of the original handwriting image layer and a recognized text layer, allowing the user to manipulate the handwriting image in the same fashion.
NotePals with Course Notetaking
Presentation slides are another "document" around which notes can be structured. This semester, undergraduates in our HCI course will be using NotePals to take notes during class. Each of the 45 students have been provided with an IBM WorkPad to take their notes on. These student notes can either be private, visible to their project team, or visible to the entire class. We have developed a web-based viewer that allows students to see their notes synchronized with the lecture slides and an audio/video recording of the lecture (see Figure 4). The synchronization is similar to the work of the Classroom 2000 project [4], but is differentiated by our emphasis on student note taking and the sharing of notes. Our evaluation will focus not only on the NotePals PDA and web-based user interfaces, but also on how students share notes and how our system changes that sharing behavior.
Figure 4. A student views their notes along with the presentation slides using the course notes viewing application.
The course notes application uses the existing NotePals infrastructure to allow a student to view his/her notes while viewing a set of lecture slides. A key feature of NotePals, the timestamping on the notes, allows us to synchronize the display of the notes and the slide presentation. We implemented a Visual Basic application that launches a PowerPoint presentation, monitors which slide is currently displayed, and records transitions from one slide to the next, as well as starts and stops in the presentation. This information is uploaded to the NotePals server.
We use a client-server architecture for the notes viewing application. The server side is a small Java-based application that runs on the NotePals web server. It manages queries to the note repository via Sun's JDBC-ODBC bridge (included in the JDK1.1). It also has requires users to log in when they first connect using the client interface. The client consists of a Java applet that makes the connection to the server and downloads the timestamping information for the selected presentation and the selected user's notes, and a JavaScript driven Dynamic HTML user interface. The interface allows users to view a single presentation, and multiple sets of notes within one view, dynamically updating the notes when users choose to view more than one set of notes (see Figure 5), or change which notes to view.
Figure 5. A student views their group's notes along with the presentation slides.
We have also been using NotePals for over a year to take notes in small meetings. These include one on one meetings between student and advisor and research group meetings. We continue to informally evaluate the design through our own use in these meeting situations.
FUTURE WORK
We have run an informal study [5] showing that users can take legible notes on the Pilot-based NotePals client in a reasonable amount of time, though for some users it takes much longer than on paper. This result led to our design of a NotePals "client" that runs on the paper-based CrossPad [6]. Users can take meeting or class notes on paper and after returning to their office quickly upload the notes to the web repository to be merged with notes taken on other devices. The implementation of the CrossPad client is incomplete at this time
We have considered adding an ink-based search mechanism [7] and possibly combining off-line handwriting recognition with a clustering algorithm to better group related notes and allow text searches on the notes. We also hope to explore synchronization with other media, such as typed meeting agendas or audio. The timestamp for each note could be used to link it to a specific event in the audio record at the moment the note was taken [8].
CONCLUSIONS
NotePals offers a lightweight, inexpensive way for people to walk away from any meeting with a low-overhead record of what transpired. It uses simple, inexpensive equipment that can be obtained by many workgroups and supports any style of meeting in any setting. NotePals’ informal, ink-based user interface combined with a zoomed view avoids many of the problems with taking notes on small PDAs, letting users focus their attention on taking notes.
REFERENCES
