Introduction to UIA: Microsoft’s Accessibility API

Introduction to UIA: Microsoft’s Accessibility API


[Guy] Hello, I’m Guy Barker, I’m a Microsoft Senior Dev in Accessibility. [Peter] And I’m Peter Frem, I’m a Senior Program Manager in Office. [Guy] And we are passionate about accessibility. At Microsoft,
we believe everyone should have equal access to information and experiences. [Peter] We want to share
what we’ve learned over the many years we’ve been
working on accessible software. Designing and programming
for accessibility is important, fulfilling, and necessary. [Guy] We also know it can
some times feel a little daunting. [Peter] So, we want to show you how Microsoft’s accessibility API
or UIA can help. [Guy] We’re going to share
accessibility best practices you can incorporate into your daily routine. [Peter] We know there will always be bugs. But in this course, we will give you tips to
avoid the most common ones. [Guy] Amazing things are happening
right now with accessibility, and we’re excited to
have you be a part of it. [Announcer] This is very exciting! By making our products accessible,
we are enabling everyone, including people with
disabilities all over the world, to do their best work! Hi, I’m Ally, and I’m here to guide
you through this course and ensure you are prepared to design and develop accessible products for everyone. Are you ready?
Let’s go. First, we’re going to learn a
little more about Microsoft’s accessibility API, called User Interface Automation or UIA and why it is essential in buliding
the best products for our customers. [Peter] Hello Peter here! [Guy] And Guy. Let’s start things off
with a quick overview. So, what is UIA? UIA, or User Interface Automation,
is Microsoft’s accessibility API. [Peter] Let’s begin by diving in to UIA and how we can use it to make
an application accessible to all users. [Guy] And that’s what it’s all about: Equal access for all users. [Peter] And that’s why we’re
so excited about this. UIA can ensure that
assistive technologies can access important and useful information about what is happening in
a particular user interface [Guy] such as whether a button is pressed, a menu is expanded,
or an item is selected. [Peter] Precisely. ATs, such as screen readers
or speech recognition programs, use this information exposed
through UIA to give the user information to help them navigate
and interact effectively with an interface. Let’s take the Bold button in the Home tab of the
Microsoft Word ribbon as an example. [Guy] Examples are great.
We want to make this as clear as possible. [Peter] That we do. That we do. Visually, there is a
square on the screen, with a capital B,
that represents a button. We call it the Bold button
because we know its purpose. The color of the button
tells me if it’s on or off. It’s part of this collection of commands and controls for the Font group
on the Home tab within the ribbon. Those visual characteristics are
represented programmatically through UIA, meaning the control type is Button,
its name property is Bold, and its accelerator key is Control + B. [Guy] How that button behaves,
or the patterns associated with that button, are also part of how UIA
makes this element accessible. [Peter] When the Bold button is toggled on, the text I enter is bold
and stays bold until I toggle it off. This is different from how the Change Case or Increase Font Size buttons
are intended to work. The Change Case button
has a menu associated with it, so it can be expanded or collapsed. When the Increase Font Size button is activated,
the font size is increased. All three items are buttons, but each button has a different
set of patterns associated with it. This button fits into the hierarchy
as part of a larger container of items: the Bold button is a child of the Font group
on the Home tab on the Lower ribbon. [Guy] See?
A place for everything and everything in its place. [Peter] Well said! If we, as designers,
PM’s, and developers, programmatically expose the
visual elements of our UI’s – by associating the right control types,
properties, patterns, and hierarchical position – we make our products accessible and aligned with the
industry standards for compliance, and when we do this,
everybody benefits. [Guy] And that’s an exciting prospect. [Peter] Be aware that the Microsoft Active Accessibility API,
or MSAA, is deprecated. So please focus on UIA moving forward. We hope this gives you a
clearer picture of UIA. But this is just the beginning, we’ll catch you later in the course
to share additional learnings about UIA. [Announcer] Before going further, I want to make sure
you’ve caught all the terms you need to know when dicussing UIA. Accessibility refers to equal access
to information within your product, regardless of what software
or hardware a person may use to interact with your product’s user interface. User Interface Automation, UIA, is Microsoft’s Accessibility API that exposes key information
to assistive technologies so a person can navigate
and interact with a user interface more effectively. Assistive Technology or AT, includes software and hardware
that can be used by people with disabilities, such as a screen reader
or alternative input device. Microsoft Active Accessibility or MSAA, is the previous version of Microsoft’s API
that is no longer supported. UIA is the way forward! [Peter] Hi again. Peter here. [Guy] Together with Guy. Let’s talk about Designing for UIA. [Peter] For applications to work best
with assistive technology, or AT, designers must peel through
the layers of what’s in a UI, and ensure UIA represents it
correctly and completely. So, with any piece of UI, you must know the type of control,
such as a button, or combobox,
you want the user to interact with. [Guy] Once you know the control type, the stage is set for
other user expectations. For instance,
when they encounter a link, they expect to be taken
to a different section on the page or to a different web page
or application. From a designer’s perspective, you need to correctly interpret
what the control type is. For example, the control type
for this element is a button which is different than the
control type of the font combobox. [Peter] Exactly. Once you know the control type, the MSDN page will give you hints
about the properties to consider. Here, for example,
is the MSDN page for the UIA button. You can see it has a
property related to its help information; you can find keyboard shortcut information, and most important the Name property, which is what a screen reader announces. [Guy] Let’s talk about the hierarchy of elements,
shall we? [Peter] We shall. The last important design element to consider, as Guy points out,
is the hierarchy of visual elements. Visually speaking, we see that
the Bold button isn’t floating; it is part of a collection of commands that form the set of tools
to adjust font style. Stepping back further, the font styles fall on the
Home tab of the lower ribbon. [Guy] Using appropriate nesting
of elements within the hierarchy allows for efficient navigation
when a person is using touch input, as one example. With Narrator on, a person can swipe to explore
the elements in the UIA tree in order. If the hierarchy, including ordering
and parent/child relationships is not clear, touch navigation becomes confusing and frustrating. [Peter] It’s really important to treat any
applicable property in its most atomic format. If you pick any one property and jam all the other relevant pieces
of information into it, the screen reader, for example, has a hard time dissecting the information and then speaking each piece
as it becomes important depending on the user’s settings. For example, if a user is not familiar
with Office or with text formatting, they may not understand what Bold does. So, the first time they encounter it,
they need details, such as its name is Bold; its purpose is to make the text bold, and its keyboard shortcut is ctrl+B. When you break down the
properties of the Bold button into its name,
help information, and keyboard shortcut, various ATs can pull each
individually at the user’s request depending on the verbosity level
set by the user. [Guy] I think you were just looking
for an excuse to say verbosity. [Peter] You know me too well. Be sure to pick the correct properties and patterns that are
applicable to that particular scenario. Also keep in mind
that some properties are only exposed if their
corresponding pattern is enabled. The Value property is an example, where the information included in that
property is not available to the AT unless the Value pattern is also supported. For the Bold button,
we use Toggle. For the Increase Font Size button,
we use Invoke. Although the Bold button
can in theory be “invoked” or activated from a keyboard,
mouse, touch input, and assistive switch, the state of the button is persistent. It’s the change in the color of the button that conveys visually if it is on or off. This aligns more closely to a
toggle pattern than an invoke pattern. [Guy] So, as a Program Manager, how would you mark up this
button for your engineers to implement? [Peter] First, dissect the screen
into individual components that you want the user to perceive,
understand or interact with. We’ll start with this template
that allows you to fill in the blanks. After you’ve identified
and dissected the pieces of the UI, ask “what is the button
trying to communicate to the user?” That is the easy part. When you know the type, you can infer the available sets
of properties and behaviors. Then, we go back to the MSDN page,
and look up what these mean. While most of these are there
for the developer’s purposes, a designer or program manager
does certain things when deciding how the button
will look and behave visually. [Guy] After all, our job here is to describe the designer’s intentions clearly to an engineer. [Peter] And clarity is king. In this case,
we say the first item is the Bold button and the second is the Font group. So first,
we need to represent a button. Next, we identify what it looks like. Then, we can snap back to the documentation to learn which properties are critical. Make sure that this button is
described programmatically in its properties. First is the Name property,
which we’ve called Bold. Then, we have some Help Text property
that might be interesting to the user. Finally, for somebody using a keyboard, we’ve described that we’ve provided accelerators to make invoking Bold much faster. That is one of the properties
available on this command. We’ve also added that in the table here. [Guy] It is the designer’s responsibility to identify whether a keyboard shortcut is necessary, so it can be documented and implemented. [Peter] We’ve now identified,
from a user’s perspective, some of the key things to know
about item number one. In the anatomy we described, it’s also important to know what happens
when I interact with it directly. This goes back to behaviors or patterns. Then I decide which makes the
most sense by asking a few questions: Is this something that would expand
or collapse after I trigger it? No, it’s not. Is this something that would
invoke after I activate it? Probably,
but invoke is fairly generic. If I ask the same question
for the toggle pattern, it seems the most applicable. At this point
we can hand this off to an engineer and they’ll understand that this element on screen needs
to be exposed as a button and that these three properties
and this one behavior communicate the button’s purpose
to a screen reader and other ATs. [Guy] Good design is essential for accessibility. Thanks, and we’ll be back
to you in a few minutes. [Announcer] Let’s recap what you need to know
when designing for UIA. Control types are well-know identifiers that can be used to indicate what kind of control a particular element represents, such as a combo box or a button. It identifies what you want the user
to interact with on the screen. Having a well-known identifier
makes it easier for assistive technology to determine what types of controls
are available in the user interface or UI, and how to
interact with the controls. UIA providers expose properties
on Microsoft UIA elements. These properties enable UIA client applications to discover information
about pieces of the user interface, especially controls,
including both static and dynamic data. The MSDN page will give you
hints about properties to consider. UIA uses control patterns to
represent common control behaviors. For example,
you could use the Invoke or Toggle pattern for a control that can be invoked
or toggled, such as a button,
and the Scroll control pattern for controls that have scroll bars, such as list boxes or list views. Because each control pattern
represents a separate functionality, they can be combined to
describe the full set of functionality supported by a particular control. The hierarchy refers
to the position a control has in relation to all controls
in an interface. For example, a ComboBox control can be the parent of both
an Edit control and a List control, and that List control will be the parent of a
set of ListItems controls. If the hierarchy is not clear, it can be confusing to assistive technology. So, why so we do all of this? Because good,
accessible design in the beginning is the key to making better,
more usable experiences. Assistive Technology or AT,
can be part of those experiences. So, let’s learn how AT comes into play. [Guy] Hello, Guy here again. [Peter] And Peter. All right, so we’ve designed for UIA
and it’s been implemented. [Guy] Well done! [Peter] Now let’s see how it all
fits together with assistive technologies. There are many ATs that use
the UIA accessibility API. [Guy] One example
is the Narrator screen reader and another in Windows is Magnifier. Both come pre-installed in Windows. [Peter] Let me show you
what Magnifier looks like. This tool relies on a subset of the UIA data to know
where a user is on screen and then magnify that region. UIA is used by Magnifier to learn
when keyboard focus changes, and what the bounding rectangle is of the control that’s gained keyboard focus. At this point I’m still zoomed into the document where I was typing and I want to
change the way the font looks. I can press the Alt key, and notice that focus
has moved to the ribbon on the top left of the screen, which is where keyboard focus is right now. As I move around,
Magnifier follows. [Guy] Very cool! [Peter] Okay,
now I’ll turn Magnifier off and turn on the Narrator screen reader. [Narrator screen reader] Document 1, Word Window.
Document 1, Editing. [Peter] Narrator just gave me the
context of where I am on screen, which is inside the document,
inside an application called Word. I’m now going to move my focus to the Bold button. [Narrator screen reader] Off. Bold Button. Alt H 1.
Control + B. Make your text bold. [Peter] You’ve heard the screen reader say which element we’re on, [Guy] that we’re on the Bold button, [Peter] that it’s currently off, [Guy] what its keyboard focus is, [Peter] and some additional instruction
of what that command actually does. The screen reader knows how to do that
based on those individual components, or anatomy, from the UIA data
that we’ve already plugged in, as opposed to it all being jammed
into a single property on that element. The screen reader looks at the control type,
in this case a button; then for properties like the button’s name, in this case Bold, its keyboard shortcut, or its accelerator keys in this case, the fact that it is a toggle-able button, and that it’s announced as on verses off. [Guy] So, there you have it. We broke down the anatomy of this button relative to how the screen reader
pulled information from it and what it says to the user. Other ATs work in a similar way. [Announcer] Earlier, you learned
about designing for UIA. Now we’d like to
show you a couple of tools that can help you find potential bugs and eliminate them in the development process. Guy and Peter will demonstrate a pair of tools, available in the Windows SDK, that they find invaluable to helping
ensure the accessibility of their projects: Inspect and AccEvent. There is definitely room for improvement
in the usability of these tools, but they offer certain functionalities that can be useful in learning about UIA. Be prepared to answer a couple of questions at the end of each demonstration. [Guy] Back again! [Peter] Thanks for staying with us
as we get into the Inspect tool. [Guy] Inspect is a very useful tool available for free from the Windows SDK. I couldn’t do my job without it. [Peter] Same here. [Guy] It can be useful for finding bugs and verifying they’ve been fixed. It’s not as intuitive or useable as it could be, so I want to explain how I use it. This tool allows Devs to learn about the programmatic representation
of the UI shown on the screen. Let’s continue with our example of the Word ribbon. First, we’ll look at some buttons on the toolbar. Inspect provides information about the UI
that lies beneath the mouse cursor or what has keyboard focus. I’m turning the Keyboard Focus Tracking option off by tapping on this dotted outline
rectangle in the Inspect toolbar but I will leave the Mouse Cursor option set. For this demo,
I will turn on an option that will highlight, with a yellow box, the UI element that Inspect is interacting with. [Peter] I also turn on the yellow box setting. I just find it easier to follow. [Guy] That’s the goal. Next, through the Options menu, I will specify the kind of UIA
information I’m interested in. Inspect can report information
about elements that are exposed through the Raw View of the
UIA hierarchy of elements – the UIA tree, but I’m usually only interested in
UIA elements in the Control View, which is a subset of the Raw View. The Control View exposes all elements
that are of interest to the customer, including all controls that they can interact with, and all text elements that provide
them with important information. Next, I’ll move to the Bold button
on the ribbon and Ta da! The yellow box appears around the Bold button. What happened in the left pane
of the Inspect window, is that the element beneath
the mouse cursor became highlighted. So now I’ll move back to Inspect
and select that element. We’ll check the hierarchy of that element
to see how it relates to other elements being exposed through the UI. This hierarchy looks fairly intuitive. [Peter] We love intuitive UI. [Guy] Indeed. Before the element is a button called
Clear Formatting and after the element is a button called Italic. All those elements are child elements
of a group whose name is Font. The hierarchy looks good, and it seems to match the
hierarchy of visuals showing in Word. Now I’m going to look at some of the properties associated with the elements
that represent the Bold button. They are showing in the right pane
of the Inspect window. Here is a Name property
with the value of Bold. [Peter] That is very important
because it will concisely tell the user the purpose of that element
in an accurate and localized way. [Guy] Exactly. Note that only the B is shown visually. It doesn’t say Bold. Word set the accessible name property
to be something useful. There are other useful properties, too. There’s the control type beneath the Name property and that has a control type of Button, so that tells you the meaning of this control; that’s very important. There are a number of other important properties,
as well. [Peter] For example, the BoundingRectangle shows him
where the control is on the screen and represents the target this
button has for touch input. [Guy] Correct.
In addition to properties, here on the right pane is information about the UIA patterns
that this element supports. The pattern is related to the
behavior of the element. As I scroll down, a number of properties are revealed
that show what UIA patterns are available. For example, can it be invoked? I find that the IsInvokePatternAvailable property has a value of false which tells me it cannot be
programmatically invoked. But then if I check the IsTogglePatternAvailable property
I see that is set to true. So this button can be controlled using the toggle pattern instead of the invoke pattern. [Peter] Any other advice we should know
about when using Inspect? [Guy] Funny you should ask. I was just going to
right click on the document and move down to the Synonyms menu. If I were to go to Inspect here,
with Watch Mouse Cursor enabled, as soon as I click into the Inspect UI the whole transient UI of the
Synonyms menu disappears [Peter] -which is not helpful when you want to examine the content of that Synonyms menu. [Guy] Not at all. So, in that case, I can have a tooltip shown when the
mouse cursor moves over an element to report specific properties that I’m interested in. To do that, I go to the Options menu,
Settings, and in the right area
I’ll scroll down to the properties I’m most interested in. By default, How Found, which is through the mouse cursor in this case, and the Name of the elements is shown. But today I’m also interested
to see whether the toggle and invoke patterns are supported by the elements. Select Okay. Next, I’ll turn on that setting to show
the tooltip and minimize Inspect. Now, I’ll move to the Bold button. There I’m told that the
name of the button is Bold, it does not support the Invoke pattern but it does support the Toggle pattern. Then I go to the Decrease Font Size button. I’m told the Invoke pattern is true and the Toggle pattern is false. If I go back to the context menu, I am told for a menu item
in the synonyms menu that Invoke is true and Toggle is false. [Peter] And that, as they say, is that. [Guy] I know this is very detailed, but these are some of the learnings we’ve had when exploring
how to best use Inspect, so we wanted to share
these tips and tricks. Now, on with the course! [Announcer] Let’s see what you’ve learned. In Inspect, which of the following
does the control view expose? More than one of the
following options may apply. A,
All hidden elements, B,
All actionable elements in the interface, C,
All elements of interest to the customer, or D,
All of the above? If you chose B,
all actionable elements in the interface and C,
all elements of interest to the customer, you are correct. The control view in Inspect exposes
all actionable elements in the interface as well as all elements
of interest to the customer. Let’s try another one. In Inspect,
if the Show Information Tooltip button is selected, it will only report the default properties. True or false? If you chose false,
you are correct. You can customize the tooltip
to reflect the specific properties that you are interested in by changing the Settings
in the Options menu. Congratulations,
you have completed this knowledge check. [Guy] Welcome back! Guy here again. [Peter] And Peter. In this video, we’ll be covering AccEvent. [Guy] Since we’ve learned about
the hierarchy of elements, control types, properties,
and patterns in the Inspect demo, let’s talk about what events
are raised through UIA. Events let the customer know
about changes happening in the UI. Let’s open AccEvent. There are two main parts to the
AccEvent interface: the Menu which offers tools and settings, and the Results window
which takes up the majority of the UI. Here are a few settings I like to customize. In the Mode menu, I check UIA events. Devs at Microsoft, like me,
care a lot about UIA events, so be sure to have that menu item checked and not the two items beneath it
relating to the legacy WinEvents. Now in that same menu
I’m going to select Settings. In here, I can check boxes
for the events I want to investigate and choose an item from the Scope pane to specify from which window
AccEvent should listen to events. The top left of this window is a check box which allows me to say whether I want FocusChangedEvents to be reported by AccEvent. FocusChangedEvents are very important
events that a UIA client app, like Narrator, will react to, so I’ll check that. [Peter] There are a lot of other events
you can highlight too, like whether a menu
or window has opened or closed. [Guy] That’s right. For example –
I’ll scroll down a bit – if I want to know whether the Name property of an element has changed I would check the Name box. If I want to know about the
toggle state of a control, I scroll down to the ToggleToggleState
item here and check that box. By doing so, AccEvent will report
three events being raised by UI: FocusChangedEvents, Name Property Changed Events, And ToggleToggleState changed events. And it’s called ToggleToggleState
because I’m interested in the Toggle State property
exposed through the Toggle pattern. [Peter] Personally,
I just like to say “ToggleToggleState”. [Guy] I know you do. On the far right of this window, I can select what properties
I want AccEvent to tell me about for the element that raised a particular event. By default, I’m told two critically
important pieces of information – the control type of the element that raised it,
and its name. So, I often leave those as the
properties I’m interested in. The final thing I do in this window is say which UI I am interested in. So, if I want to know about all the
events raised by any UI on my desktop, I would select Desktop in the
middle pane labeled Scope. Right now we will just look at events being raised from the Document 1 Word window. [Peter] And if no element is selected in this middle pane, AccEvent will send you an error message. [Guy] Right. I’ll leave the default check boxes
set up to say I want events reported in order to learn about events that are raised by the Document 1 Word window
and all its descendants. So, I’ll select OK. Now I go back to the main AccEvent window, select the Events menu,
and say Start Listening, which will mean
AccEvent will report UIA events that are being raised by UI. AccEvent always reports a
few initialization steps in the beginning, but now it’s waiting for events to be raised. It would be rude of me to make it wait. [Peter] Would it? Most likely. So, if I switch into the
Document 1 Word window, AccEvent reports that a
FocusChangedEvent was raised and it was raised by an
element whose name is Document 1. The control type of that
element is a Document control type, which tells me that that
Word has now raised a FocusChangedEvent to say that keyboard focus has
moved into that window as I selected it. So, I’ll use the keyboard and tab through the elements on the ribbon. As I do, AccEvent reports
that there are a number of FocusChangedEvents being raised. In fact, one is being raised as I tab. If I press the Tab key now, I expect a FocusChangedEvent to be
raised by the Format Painter button, because that tells me that
keyboard focus has moved to that button. That event also allows a screen reader,
like Narrator, to tell the user that they have
reached the Format Painter button. I’ll press Tab now,
and there we have the FocusChangedEvent, meaning that a FocusChangedEvent was raised by the Format Painter element. [Peter] Well, that was “eventful”. [Guy] Yes, yes it was. Further up in that list of events raised, there was a PropertyChangedEvent. A Tab item on the ribbon
raised a PropertyChangedEvent, saying that the name property had changed to Home. That happened when I hit
Alt-H to move keyboard focus to the ribbon. I’m going to keep on tabbing
until I reach the Bold button. As I did that, I got a number of FocusChangedEvents as keyboard focus moved
between the various buttons. Now that keyboard focus is at that button – I’m going to use the keyboard to
toggle the state of that Bold button. When I did that,
a PropertyChangedEvent was raised by the UI. [Peter] Yes, and from the information
in the AccEvent window, We can tell that the control that
raised it was a Button control, Its name was Bold, The property that changed was on the toggle pattern, and it was a toggle state property. [Guy] AccEvent reports that the
value of that property is now on, which matches the visuals on the screen. When I press the spacebar to toggle the button, the button looks like it’s been depressed. If I do that again, AccEvent tells me that the
focus moved into the document, so I’ll move keyboard focus back to the ribbon, release the Bold button by hitting Alt-H-1… and there you go! [Peter] So now, AccEvent reports that the PropertyChangedEvent again
is raised by the Bold button, and this time the toggle state property has a value of off. [Guy] Exactly. [Peter] Wow, thanks Guy,
that was a great demonstration. A lot of information here,
but well worth it. [Guy] Yes, hang in there everyone, this is important work and the dividends are that we make
products that are great for everyone. [Announcer] Let’s see what you’ve learned about AccEvent. You’ve been assigned to a bug
that reports that after activating a button, keyboard focus is not where
the user expected it to be. You use AccEvent to investigate. What setting will you enable to make sure
AccEvent notifies you when focus changes? A,
FocusChangedEvent, B,
GridItemRow, C,
ToggleTogglePattern, or D,
none of the above? If you chose A,
FocusChangedEvent, you are correct. FocusChangedEvent will reveal
what events are being raised that impact the keyboard focus for the user. Let’s try one more. Using AccEvent,
you learn that when you tab to the button, there is no FocusChangedEvent raised. What is likely the reason for that? A,
the button is not using a standard button control type, B,
the user is doing something incorrectly, C,
the name on the button is incorrect, or D,
the hierarchy is too complex? If you chose A,
the button is not using a standard button control type, you are correct. When using a standard control, you inherit a good amount of
programmatic accessibility. When using custom controls, you become responsible for
making your control accessible, which takes more time and effort. In this case, the person who built that button, started with a custom control and did not do the work to
make it programmatically accessible. Congratulations,
you have completed this knowledge check. Are you ready to put this all together? Now, Guy and Peter will demonstrate
how many of these concepts come together. One thing to know as you watch the next video: Implementation will differ based on
which UI framework you are using. [Guy] It’s us! Guy, [Peter] And Peter! And this segment we are calling,
“How to do it right”. It shows you how to make sure that the customer has a great experience interacting
with your product through UIA. [Guy] Earlier you heard that PMs consider the ways an element shown on the screen
might be represented through UI automation. So, let’s look at this button which shows the text A with a red-filled rectangle beneath it, that visually indicates text color. The PM knows that the color
must be conveyed programmatically, so users of assistive technology can access it. So, the PM chose a useful, localized, and accessible name:
Font Color. Often, using a standard control
supplied by the UI framework will give developers a jump start at fulfilling the requirements the PM set out. [Peter] So, you actually prefer standard controls. [Guy] I do because when you
start creating custom controls, you don’t know how much
extra work you may create, too. And it could be a lot. So now we will switch to Visual Studio and see how that button is actually implemented. There is a standard button control
being used to present visuals. And inside that button, there is a text block whose text is A with a filled rectangle beneath it. Switching back to the app, we’ll point Inspect at that button and hover. Inspect expanded the UIA tree and reveals where the button lies
in the UIA tree of elements. That button is contained within a window element, whose name is ValueButton, and that window element is contained within another window element of the same name. [Peter] Does that button contain any child elements? [Guy] To find that,
move over the text shown in the button, and Inspect reveals a separate element
being exposed through UIA. When I see that, I’m really interested in what view
of the UIA tree I am looking at. So, I go to the options menu
and confirm that I am looking at the Control View of the UIA tree. That means that both the text block
and the button are being exposed through UIA,
through the Control View. By doing so, I’m saying that both elements
are of interest to my customer. [Peter] But as a customer, I consider that whole control
to be a single element of interest – a button. [Guy] Fair enough. So, I will remove that text block element from the Control View of the UIA tree, so when the user interacts with the UI they only find the button, not what’s contained within it. To remove an element from the Control View, I’ll switch back to Visual Studio
and I’ll edit the XAML. I’ll use AutomationProperties. AccessibilityView=Raw. I’ll run the app again, start Inspect, point it to that button, and select the A text. Now the text block is not found, and when I try to expand that element there is no element being exposed. That means any screen reader that does hit testing over the UI
won’t find that text block. [Peter] That seems much better for the user,
right? [Guy] Right, indeed. So, let’s look at the right side
of the UIA window to see the properties being exposed by this element. The control type is being exposed as
“Button,” which is great. The BoundingRectangle shows me
where and how large the control is and it also shows many helpful properties
such as IsEnabled and IsKeyboardFocusable. [Peter] Overall, that looks correct. [Guy] It does. However,
one essential property is not being exposed. [Peter] Which is? [Guy] The Name property. The Name property must be accurate,
concise and localized, and convey the meaning
of this button to the user. To set an accessible name I’ll add an x:uid to the button
and call it FontColorButton. Then, I can add localized resources, which will be referenced by this button. I’ll then open up the strings and go to the localized string
resources for the app. Earlier, I added a localized string
resource of Font Color as the accessible name of this button. This is very similar to how I would
set a localized visual string that is exposed on the button. There, I would do FontColorButton.
Content, but here, I set AutomationProperties.Name. Now, I’ll run the app again, start Inspect, and point it to the button. It shows the accessible name of Font Color which completes the set of properties
that the assistive technology needs. [Peter] Well…look at that. One simple change
and the properties are sorted out. [Guy] Next, my PM told me that a
value for the button must be exposed through UIA, which reflects the current color
associated with the button. That value is exposed through the
UIA Value Pattern in a property called Value. On the right side of Inspect, I will scroll down to determine if this control supports the Value Pattern. [Peter] And its property is False. [Guy] True. I mean correct; the property is indeed False. Since I need the Value Pattern exposed here, I will enhance the accessibility
using the XAML UI framework. [Peter] Very convenient. [Guy] Now, I’ll close Inspect and the app
and switch back to Visual Studio. I’ll add a new control that
will derive from the button control, and call it a ValueButton. Because it derives from a button control, it’s automatically as accessible as a button control. I’ll then add another new class called
the ValueButtonAutomationPeer and that will derive from a XAML class
called ButtonAutomationPeer, which provides all the
accessibility for a button control. In order to tell UIA that my new
ValueButtonAutomationPeer is to be used with my ValueButton, I override the method OnCreateAutomationPeer and return an instance of my new
ValueButtonAutomationPeer. When I run the app, my new ValueButton has all the
accessibility of a regular button. But now I want to enhance that
accessibility to support the Value Pattern. First, I will have my new AutomationPeer inherit from the UIA IValueProvider interface. I’ll then override a method called GetPatternCore, which is called by UIA to ask, ‘Is this pattern supported by this control?’ I’ll add a special check to say ‘If UIA wants to know if the value pattern is supported,’ then I’ll say ‘Yes it is,’ and the object that implements
that pattern is this AutomationPeer. [Peter] At this point you haven’t added support for it,
correct? [Guy] Yes. So next, I go to MSDN, find the various properties
and methods associated with this pattern, and add support to them in this AutomationPeer. There is a method called SetValue, which allows a UIA client to programmatically set the
current value on this control, and then two properties for
‘Is the value read only?’ and in this case we’ll say that it’s false, and also a property to get the current value. While I’ve hard-coded the value to be Red here, that wouldn’t be the case in a shipping app because the string would be localized. Having done that I’ll now run the app,
and again, I’ll point Inspect to this new control and see what patterns it now supports. [Peter] This is a lot of information! [Guy] Yes. But it all becomes clear as you go! Ok, so I’ll scroll down
and before we get to the ValuePattern let’s take a quick look at the
IsInvokePatternAvailable. That property is true, and that is because the ButtonAutomationPeer that was used to create the new
ValueButtonAutomationPeer already supported the invoke pattern. Scroll down further and you’ll notice that the IsValuePatternAvailable
property is now true as well. Next let’s take a look at the properties being exposed through the value pattern on this control. There are two properties being
exposed through the value pattern, one IsReadOnly, which is false, because we specifically hard-coded that in the code, and the other is the current value,
which is Red. Now let’s go to the
SetValue method in the Actions menu and have Inspect programmatically call that method in the code we just wrote and set a particular value. So, let’s type “Blue,” and execute. There we go, the value is now Blue. We have now updated, or enhanced the default accessibility of this button
to support the ValuePattern. [Peter] So, quick review: we’ve seen the hierarchy, the properties and the patterns, but one last thing we should check here is events associated with that control. Am I right? [Guy] You are! By default, it’s a standard button control base, so a lot of useful events will already be raised. However, I’ve just enhanced the accessibility of this control
to support the ValuePattern. That means I’ve got to be sure when the value of the Value property changes, that the event is raised to allow the screen reader to know that the value has changed. So, I’ll reduce the size of the Inspect window here and bring up the AccEvent tool. I’m going to change the settings in AccEvent to say I’m interested in knowing when the
Value property of this element changes, so I’ll clear existing events that are being reported here. Now I’m going to have the ValueValue event reported. I’m going to say I’m interested in events being raised from the app window. By the way, it’s called ValueValue here because it is the Value property of the Value pattern. [Peter] Hmmm… “ValueValue”. Doesn’t have the same ring as say, “ToggleToggle”. [Guy] You have a point. So, I will listen for events and I’ll switch to the app. I can tab to the button control and have FocusChangedEvents raised by default, and I’ll point Inspect to this. I will say currently the value of this control is Blue, and I’m going to change that to say Green, hit Execute, and close it. When I did this, AccEvent reported no
PropertyChangedEvents were raised. That’s very important because it means as the value of that property
changed from one color to another, there was no event raised to let a
screen reader announce it to a user. So, there is one more step to have an event raised to give a screen reader
a chance to react to the change. So, I’ll close down the app and take us back to the
set value method we used earlier. This is where a UIA client calls in to say ‘I want the value of this element changed.’ We did change the value, but we need to take one additional step, to have an event raised to let UIA, and UIA clients, like a screen reader, know of the change. So now we will run the app, and I’ll run Inspect and AccEvent. Because Inspect remembers the previous settings, I just say
‘Report events from this window.’ Then, I go back to the button, look at its current value, (it is probably Red,
the default for this demo app) and say ‘
change that value from Red to Yellow’ again. Execute, and close. AccEvent then reports a number of
FocusedChangedEvents as before, but also a property changed event. It says ‘the sender of this event is a ButtonControl whose name is Font Color, and it’s a Value property of the
Value pattern that has changed.’ And its new value is Yellow. Now, a screen reader can detect
the change in the value of that button, and let the screen reader user know. [Peter] We are convinced that the
programmatic accessibility of this control is a great experience for the customer. In other words, we did it right; just like we said at the beginning. [Guy] Thank you for joining us
for these demonstrations everyone. [Guy] The adventure is ongoing. We look forward to further developing UIA as Microsoft’s accessibility API
to not only make products better, but to help make the tools we use more effective. [Announcer] We’ve covered a lot of information in this course. Let’s take a few minutes
to see what you have learned. You want to include a button
in your app that has a unique design. What initial steps should you take? More than one of the following options may apply. A,
verify if there is a way to build that button by adding visual enhancements
to a standard button control. B,
if a standard control is not possible, estimate the time it will take
to create custom buttons and make them accessible. C,
if a standard control is not possible, discuss with your lead
the extra work that will be required. Or D, go ahead and make the custom button
and hope that it doesn’t get a bug. If you chose A, B, and C, you are correct. Standard controls inherently
include programmatic accessibility. When building a custom control, you become responsible for adding the
accessibility properties and behaviors, including keyboard access. This adds time to the development of your UI and runs the risk of unexpected bugs during testing. Congratulations!
You got the approval to build the buttons. Now, you need to ensure that the visual characteristics are represented programmatically through UIA. Which characteristics are essential? More than one of the following options may apply. A, patterns, B, controls types, C, properties, or D, where it is in the hierarchy. If you chose all the options,
A, B, C, and D, you are correct. All of these characteristics are essential
for assistive technologies to work properly. You completed the development of the button. Now, you want to use AccEvent
to check that the UI is being correctly relayed to the customer. When you first open AccEvent,
which settings should you check? More than one of the following options may apply. A, UIA events, B, WinEvents (in context), C, WinEvents (out of context), or D, FocusChangedEvents. If you chose A, UIA events
and D, FocusChangedEvents, you are correct. These are the first items to check
when starting AccEvent. They are essential pieces of information that must be reported to the user. You made good use of your time deciding
on the control types and properties, and want to be sure you haven’t
introduced bugs in the process. What’s you next step? Again, more than one of the
following options may apply. A, use the Inspect tool to
confirm the design of the buttons matches the representation in UIA. B, use an assistive technology, such as the Narrator screen reader, to test the user experience with your UI. C, use AccEvent to ensure the
RotationPropertyChangedEvent is invoked. or D, use the Inspect tool to ensure that the button is placed
correctly in the hierarchy. If you chose A, B, and D, you’re correct. Inspect is used for finding and fixing bugs as well as learning about the
programmatic representation of UI that is shown on the screen. Congratulations,
you have completed this knowledge check. You’re likely to have questions as you start using UIA. We’ve put together answers
to some frequently asked questions to help you avoid common pitfalls. Is there a UIA property called ‘role’? Role is often used to refer to an element on the web, as referenced in the Web Content Accessibility Guidelines, or WCAG. Although technically there is a “Legacy I Accessible Role” in UIA, we should pay attention to the
Control Type property in UIA, Control Type best aligns with the
modern concept of “role.” How do I know if I’ve used UIA correctly? Verify all your control types
are using the available patterns or behaviors from the UIA spec in
MSDN for that kind of control type. As you gain confidence in UIA design
and implementation, it might be helpful to also test
with an assistive technology, such as a screen reader, to get a sense of what experience
you are building for the customer with your choices in UIA implementation. Help! I just can’t get UIA to do what I want it to. What can I do? UIA is still evolving, and more functionalities will be added. In addition to UIA, the framework in which you are building may have limitations around accessibility. Check out the Microsoft Windows UI Automation Blog for guidance and examples. I’ve never used assistive technology
before for testing, where should I start? There is a whole ecosystem of
assistive technology that can run on Windows. To determine which type of
assistive software to begin testing with, first learn what kinds of
assistive tools your customers use. There is a suite of assistive tools
in the Ease of Access Center that comes pre-installed on Windows. A product, like Narrator, will help you get a sense of how UIA
is being handled in your app. I still need help with UIA, where do I go? The Windows Developer Support site
is a great resource to leverage. I’d like to learn more about accessibility,
what is available? The Microsoft Accessibility homepage
is a great place to start. And thank you for wanting to learn more – we knew you were smart! Let’s quickly summarize what has been covered. 1. UIA is Microsoft’s Accessibility API that ensures assistive technologies
can access important information about how a person can navigate
and interact with an interface. 2. The key characteristics
of UIA are Control Types, Properties, Patterns or Behaviors, and Hierarchy. 3. You should always prioritize
standard controls over custom controls. 4. Designing with the user
in mind from the beginning will ensure a seamless workflow, less bug remediation and better, more accessible products. Thank you for learning more about
Microsoft’s Accessibility API, UIA! We hope you’ve learned a few best practices
from Guy and Peter and we’re excited about the experiences you will go and create with that knowledge!

3 thoughts on “Introduction to UIA: Microsoft’s Accessibility API”

Leave a Reply

Your email address will not be published. Required fields are marked *