Using the new Office with touch

Editor’s note: Windows
8 provides a number of platform capabilities for enabling highly responsive
touch support in applications, ranging from hardware accelerated graphics and
improved touch targeting to the a new app platform that makes it easy to build
touch-optimized Windows 8-style apps. The new Office takes advantage of these
to deliver great new touch-based experiences on Windows 8. Clint Covington, a lead program manager for our User Experience
team authored this post. We’re hearing reports of IE10 users not successfully viewing the videos. If you are unable to play the videos, please visit the Office Next page on Facebook. http://www.facebook.com/MicrosoftOfficeNext . We’ve also updated the post to point to higher quality videos on Office channel on YouTube, and these are compatible with IE10 (both modes).

Clint Covington

On Monday in San Francisco we took the wraps off of the new
Office’s touch experience designed for Windows 8. We showed the new touch-optimized
Windows 8-style app for OneNote, and we showed how we’ve touch-enabled Word,
Excel, PowerPoint, Outlook, and other apps on the desktop. The new Office is
designed for a great experience whether you’re sitting on a couch with a
tablet, or at a desk with a mouse and keyboard. It makes common tasks fast,
fluid, and intuitive, while still enabling the rich capabilities required to
create high-quality documents.  In this
post I’ll walk you through the thinking, engineering process and design
framework we used to reimagine these experiences for touch.

 

Our vision for touch

When we started planning the next version of Office, it was
clear that touch was going to be a big part of how people used Office in the
future, and we were excited about the opportunity to add a delightful new
dimension to our apps. Hardware was evolving rapidly to enable new types of PCs
and mobile devices, and along with them new styles of interaction, new physical
postures, and new usage locations. We believed people would continue to use
Office at their desks, and type on physical keyboards at a desktop or laptop,
but would also start to use tablets and hybrid laptop/tablet devices in a
variety of other situations: leaning back in a chair, leaning forward on a
train, bus, or plane, sitting on a couch, lying in bed, standing and holding
with two hands, or walking down the hall.

We analyzed what kinds of tasks were comfortable to do in
each of the postures, and what kinds of input were best for different kinds of
tasks. For example, a physical keyboard is optimal for large amounts of typing
(still significantly faster than an on-screen keyboard for most people). A
mouse is optimal for precise targeting, and touch is great for broad strokes
such as scrolling and zooming. In some postures, such as standing, touch is the
preferred input, while the efficiency of typing at a desk is hard to beat. Each
posture and input characteristic is great for some jobs and not as good for
others. We wanted to make sure Office apps felt intuitive, natural and
comfortable as across different postures and different kinds of input.

In parallel, Windows was introducing deep investments in
touch, ranging from new touch digitizer requirements for touch-enabled PC
hardware, through new touch drivers, new platform APIs, and a new user
experience. This platform provided us with the core capabilities to deliver a
natural experience across our applications and the new usage postures. (As a
side note, many of Office’s new touch capabilities will work on touch machines
running Windows 7, but improvements in the underlying platform in Windows 8
make the experience substantially better, improving everything from the touch
targeting accuracy – making it easier to tap the button you meant to tap – to the
button sizes on the screen, to the speed and responsiveness of the touch
feedback.)

We’ve invested in touch in two ways for the next version of
Office. The first is that we’ve begun to build new versions of some of our apps
(OneNote and Lync) specifically tailored for the new experience in Windows 8.
For these, we deliberately started from scratch to design new versions that
were touch-first. The second is that we’ve touch-enabled the familiar desktop
versions of all of the Office apps. These continue to be designed first and
foremost for mouse & keyboard, and to preserve the features & layout
you’re used to, but now support touch throughout the experience. This dual
approach allows you to immediately get the full power of Office on a touch
machine with no relearning required, along with new experiences that embrace
touch from the ground up and deeply integrate with the new experiences in
Windows 8.

The scope of our vision meant we would be making broad changes
across all of the Office applications. We started by identifying the heavily
traveled paths through each app and using these to focus our efforts.  Some examples were working with mail, sending
an instant message, reading a document, editing a spreadsheet, and presenting a
slide deck. We wanted to ensure that each of these felt great end-to-end using
only touch, or a mixture of touch, mouse, and keyboard.

Guidelines for touch

The next step was to develop a common framework for how we
would enable touch across Office. After comparing our current experiences with
where we wanted to go, we came up with the following areas of investment:

  • Touch responsiveness
  • Touch targeting
  • Selecting text & objects
  • Typing
  • Commanding

We wrote detailed guidelines for each of these areas and
established minimum bar expectations for how the work should come together. We
worked closely with people across the company to make sure the internal touch
language is consistent across Microsoft experiences and cultivate learning
transfer. We built out scorecards for each scenario and identified areas of
work that needed to happen to achieve a consistent, responsive and delightful
experience that leaves people touching with confidence. Here is an example of
one of our early scorecards for the work with mail scenario:

Let’s take a look at each of these guidelines.

Touch responsiveness

When you touch something, it needs to immediately respond. Content needs to “stick to your finger” when
you pan and zoom. The UI needs to respond with inertia and bounce when flicked
or reach the top/end of sections. Pictures need to immediately resize as you
drag your finger, and text needs to reflow around it to give the user a sense
of physical realness to the action.

To build an experience where content sticks to a finger, we
had to refactor much of the document surface to use a compositor where content
is rendered into images and moved around on the screen with animations. This
required us to migrate away from GDI to modern hardware accelerated graphics
services. The result of this work is an experience that feels natural and
responsive. Here are some examples:

 

 

Targeting

Targeting refers to successfully touching the thing you’re
trying to touch. Targeting is almost entirely about raw physical size. Fingers
are much bigger than mouse pointers or pen tips, so things need to be
physically larger on the screen in order to be comfortable to touch. The
Windows team did extensive research to develop guidelines for hit target sizes,
which we used throughout our designs.

The new Windows 8-style apps for OneNote and Lync were
designed from scratch, so we were able to ensure that every part of the
experience was easy to touch right out of the gate. In the desktop apps, many
of the typical hit targets designed for mouse and keyboard did not meet these
guidelines, so we had to figure out how to resize them. We identified two types
user interface elements that needed to be adjusted larger:

  • Fixed parts
    of the user interface that are always visible, such as the Quick Access Toolbar (QAT), status bar, ribbon, and folders in
    Outlook
  • Contextual
    parts of the user interface that appear
    in response to a user action, such as context menus and the Minibar

 

Fixed user interface elements

Many fixed user interface elements fell well below the
minimum size to be touchable. It was not possible to reliably touch common
commands like the Save button in the QAT. So we added a new feature called
Touch Mode that increases the size of fixed UI. Touch Mode is automatically
turned on for properly configured tablet machines. It increases the size of the
QAT, Ribbon tabs, adds spacing around small buttons in the ribbon, increases
the height of the status bar, turns on the Outlook touch triage action bar and
adds space to expanded folders in Outlook. Here is an example of a portion of
the Word ribbon with Touch Mode on and off:

Touch Mode Off:

Ribbon with Touch Mode Off

 

Touch Mode On:

Ribbon with Touch Mode On

You can turn on Touch Mode by clicking on the QAT overflow
and selecting the Touch Mode icon.
The icon then will show up in the QAT and can easily toggle Touch Mode on and
off.

 

 

Contextual user interface elements

A lot of Office’s contextual UI (menus, context menus, color
pickers, mini-tool bar, etc) contains small hit targets as well. Since
contextual UI isn’t visible until you take an action, we can decide whether to
show a touch or mouse-optimized version based on whether the action was a touch
or a mouse click. So contextual UI can always be the right size regardless of
whether you’re in touch mode. For example, if a context menu was invoked through
touch, we spaced items in the menu to make it more comfortable to touch. Here
is an example of how the color picker renders differently for touch verses the
mouse:

Mouse Color Picker       Touch Color Picker

For shapes, charts and pictures we increased the size of the
grab handles and used Windows touch targeting APIs to make it easier for users
to select and resize objects.

Mouse Grab Handles    Touch Grab Handles

New Windows 8 APIs tell us the shape of a finger on the
screen and the first contact point. If the first contact point is not inside
the grab handle, we have the opportunity to detect if a finger overlapped it
and ensure the selection is successful. The end result is a finger feels more
precise and people miss-tap less. These touch targeting APIs are used across
Office to help users touch with confidence through more accurate determination
of user intent for sloppy or imprecise touches. See the Touch
hardware and Windows 8
post on the Windows
8 Engineering blog
for more detail.

Positioning an object by dragging

One special case of targeting is grabbing and object and
dragging it. When you are simply dragging something onto another object, for
example, dragging an e-mail onto a folder in Outlook, the difficulty of
targeting the folder while dragging isn’t much different from trying to simply
tap on it. (It’s actually a bit easier.) However, when you’re dragging an
object in order to position it, it’s hard to get the same precision you get
with a mouse. To help with this, we’ve added guidelines and snap points that
appear automatically when moving or resizing pictures, shapes and other
objects. This makes it easy to line them up with margins or other objects.

 

 

Text and object selection

One of the most fundamental actions in Office is selection
of text and objects. Mice and keyboards make selection easy with precise
targeting & modifiers such as the shift and control keys. A finger lacks
the precision required to do these operations efficiently, so we worked with
the Windows team to develop a series of selection patterns for text and
objects.

We added new text selection handles to Word, PowerPoint,
Excel, OneNote, Visio, Lync and messages in Outlook. Here are a couple of examples
from Word and Excel:

These are the same text selection handles you see across
Windows 8 in apps like the new modern browser.

To select multiple objects, we followed the Windows
guidelines for cross-slide to select multiple slides and drag and drop to
rearrange. You can easily select multiple objects in PowerPoint by keeping one
finger down and tapping on other shapes and pictures.

 

 

Typing

Typing is a part of many scenarios in Office. Windows 8 made
a large investment in its on-screen or “virtual” keyboard. In the new Windows
8-style apps for OneNote and Lync, the virtual keyboard appears and disappears
automatically, leveraging APIs provided by Windows.

In the desktop apps, we needed to do some more work:

  • The keyboard didn’t appear automatically when
    you needed it
  • It sometimes covered what you were typing
  • You had little space left to work while it was visible

Invoking the keyboard

With Office 2010 and Windows 7, people have to manually
invoke the keyboard through clicking on the keyboard icon in the Windows task
bar. Windows 8 provides new desktop APIs that allow Office to hide and show the
virtual keyboard automatically.

Keyboard covers what you’re typing

Few things cause more frustration in touch than when the
virtual keyboard shows up and overlaps content or the area where you are
typing. It is even more frustrating when as you type, the cursor scrolls under
the keyboard. Windows 8 includes new desktop APIs that inform our applications
when the keyboard is coming and going, as well as where it is on the screen.
This lets our applications scroll content out of the way and keep the cursor in
view as you type.

Working space when the keyboard is visible

We found there was little space to work with content when
the ribbon is expanded and the virtual keyboard is up. This leaves users
feeling cramped and frustrated. To solve this problem, we default the ribbon to
collapsed mode on tablet machines. This gives users more space to work. We
introduced a number of subtle refinements to the interaction model of the
ribbon to make it work even better in collapsed mode. You will notice that it
now is possible to execute multiple commands at a time (such as Bold, Italic
and Underline). Yet, when you execute a command that is highly unlikely to be
followed up by another ribbon click, it goes away (e.g. Insert Table).

We also build a full screen view that allows users to hide
the ribbon and status bar so they can focus on content and see even more lines
of text or rows of cells without the chrome taking away from valuable content
space. We will provide more details about full screen view in a subsequent
post.

Minimize the need to type overall

The virtual keyboard is not as efficient for typing text as
a physical keyboard. Many of our scenarios and new postures required users to
use the virtual keyboard to complete a job, and we wanted to reduce traffic to
the virtual keyboard where ever possible.

Common patterns include suggestion menus and most recently
used (MRU) lists that remember previous input. For many years, applications
remember paths to recently opened documents. This used to work well, as people
primarily used one computer. With the use of Smart Phones, Tablets and the
cloud, people need to access documents across different machines, at home and
at work. If you are connected to the Office service, your document and common
locations MRU will roam across all of your devices. This significantly reduces
traffic to the file open and file save dialogs which invariably involves typing
in hard to remember, archaic file paths.

Excel’s new Flash Fill is another feature that saves you
time because Office does the typing for you. Here’s how it works:

 

 

Commanding

The final area of focus for touch is the command experience.

In the desktop apps, it was nearly impossible to complete
any of our scenarios without using the Ribbon, Backstage, Context Menus or MiniBar.
Additionally, we know from the Customer Experience Improvement Program that
many commands are executed through the keyboard. Some of the most common
commands like Cut, Copy, Paste, and Delete are most frequently executed through
the keyboard and not the application’s command interface. Here are a few of the
guiding principles we came up with as we refined the commanding experience:

  • Common commands are easy to find and fast to use
  • Users locate commands in consistent places
    between touch and mouse/keyboard experiences
  • Frequent keyboard actions
    (cut/copy/paste/delete) are easily accessible
  • The user experiences feels familiar and
    touchable

As we evaluated our scenarios, we found that most would be
accomplished with a small subset of commands. When people use Office with touch,
we expect them to work on different tasks than sitting at a desk with a mouse
and keyboard. We also found for many contexts (selected text, picture, shape,
table, etc) a small number of commands make up the majority of actions. As
mentioned earlier, the vertical real estate becomes a scarce commodity. All of
these insights lead us to want to provide users with quick access to common
commands while keeping enough vertical real estate to work with their content.

In addition, in designing our new apps for Windows 8, we
aimed to be consistent with the goals and design language of all of the other
Windows 8 apps that are being developed and deliver an experience optimized to
help you get your work done. Most importantly, we expect you to be able to
focus on your work and we want the Office user interface to get out of the way.
Windows 8 is designed so that apps have control of every bit of real estate on
the screen and we want to pass that real estate on to you.  This gives you
the room you need to work – to read, to type, to move around and see your
content.  When you do need to take an action, then the commands that you
need are just a click, tap, or swipe away.

MiniBar

Office 2007 introduced the concept of a MiniBar

a small strip of commands that show up when you select text or certain objects.
The minibar was designed to provide efficient access to frequent commands. For
Office 15, we extended the minibar to support our touch scenarios.

In the desktop apps, we changed it to show up when users tap
on selected items like text, cells, images and shapes. We added common keyboard
commands such as Paste, Cut, Copy and Delete to decrease traffic to the virtual
keyboard and help people accomplish scenarios much faster. Here the MiniBar for
a picture:

For second-tier commands, there is also a contextual menu
drop down from the MiniBar. These are the less frequent commands, but makes the
common right click actions with a mouse and keyboard also available with touch.
When the contextual menu shows, it does not repeat commands already on the
minibar (such as cut, copy and paste). Here is the same picture with the
context menu expanded:

In-Canvas Commands

Communicate with others is one of our most important
scenarios for tablets because we expect users to spend lots of time in Outlook
triaging mail. We found for mail there were a small set of commands that
dominate usage. These frequent commands include Reply, Reply All, Forward, Delete,
Move to Folder, Flag and Mark as Unread.

Originally we exposed these options in the mail message list.
As we started using Outlook on tablets, it became clear the posture of the
usage was different. It was extremely difficult to target a delete button in
the middle of the screen when in the preferred lean forward posture while
holding the device with two hands. In this position it is much easier to use
thumbs to target real estate along the edges.
Here is how the thumb bar works in Outlook:

 

Radial menus

Your tools at your fingertips. That’s the goal behind
the radial menus that
are available while you are typing and editing in the new OneNote app for
Windows 8. The commands that we expect you to use most while editing, we
make available next to your content.  While
you are working, your commands stay tucked behind the menu.  But when you
need to format your text, for example, just click or tap on the radial menu and
the commands you need appear:

 

The radial menu is designed so that the most common commands
you need are available immediately.  Then if you need more commands, you
can just swipe your finger out from the center of the menu or tap on any of the
arrows around the menu.  For instance, more formatting commands are
available right there next to bold:

 

You’ll also notice that when the commands are tucked away,
the radial menu changes based on what you are doing.  For example, if you
are typing and need to insert a table or a bulleted list, you’ll see:

However, if you are working with images, you’ll see:

Under each menu are the commands or tools that you’ll need
at that time. And you don’t even need to expand the menu to select the 8 most
common actions. You can just swipe the radial menu in the direction of the
command you want. For example, if you have some text selected, swiping right on
the collapsed radial menu will bold text, swiping up will copy, and swiping left
will undo. They’re like keyboard shortcuts for touch. These quick swipes are just
one more way radial menus can make touch interactions more fast and fluid,
without getting in the way of your content.

 

App Bar

Finally, like all other Windows 8 style apps, we expose the
commands that you need to navigate within Lync and OneNote in an app bar at the
bottom of the screen.  Just swipe in from
the edge of the screen or right click (or press the Windows key + Z) and an app
bar slides in:

In this screen shot from OneNote, you can see that you can
move back and forward through your notes, bring up your list of notebooks or
pages, add a new page, or give us feedback on the app (please do!).

The app bar will also change based on what you right clicked
on.  For example, if you right click on a section within an OneNote
notebook, additional commands will appear on the left side of the app bar.
Now you can also rename or delete the section or copy a link to that section:

 

Summary

We are excited about the opportunity to enable a whole new
set of usage postures for Office, from the bus to the plane to the couch to the
hallway. We are excited how with touch common tasks are fast, fluid and
intuitive while still enabling the rich capabilities required to create
high-quality documents. We hope you enjoy using the new Office as much as we
did building it. There is still a bit of touch polish work left but I encourage
you to download the preview, install
OneNote MX
, and please give us feedback. We look forward to your comments!
Thanks