Programming your home automation with the Google Assistant (Behind the Actions, Ep. 3)

Programming your home automation with the Google Assistant (Behind the Actions, Ep. 3)

I’m Neto, developer advocate on the
Actions on Google team. And I’m here today for one more
“Behind the Actions” episode. And today, we have a new guest. Hey, Daniel, how are you doing? DANIEL MYERS: I’m doing well. How are you? NETO MARIN: And
Daniel is my teammate. So what do you do in our team? DANIEL MYERS: So I’m a developer
advocate here at Google, specifically on the Google
Assistant and Actions on Google team, where I work
a lot on various Smart Home and IoT-related aspects. NETO MARIN: So you can
see he has cool suit. He do as the cool industry. So [INAUDIBLE]. [LAUGHS] Kidding. So, OK, but what is
this weird stuff here? What are you going to
present to us today? DANIEL MYERS: Absolutely. So this is a Raspberry Pi. It’s actually a Google AIY
Vision Kit where we will be modifying this off-the-shelf
device to enable Google Assistant controlling to where
you can give it various smart washing commands, like “start
my cycle,” “run cycle,” turn it on and off– things like that. NETO MARIN: Cool. So if, for example,
I’m running behind, I’m running late
to catch my bus, and I forgot to start
my washer machine, I could send some command,
and even from my shuttle, say, hey, forgot. Start my laundry. DANIEL MYERS: Exactly. Exactly. You know, well,
part of the reason why I chose smart
washing machine today is because it’s a little
bit more complicated than just a smart
light bulb, right? We still have the on/off you
would have with a light bulb, but you also have things like
modes and various other traits that you get exposed to when
developing a smart washer. NETO MARIN: Oh, yeah. It’s a good fit for our
“Behind the Actions,” right? I’m going to show some deep
dive on this structure. But, yeah, before we start,
can you show us how it works? DANIEL MYERS: Absolutely. So I have this already
setup with my account. I have it set up with
my Google account. Everything’s already
been deployed. And so what I’m going
to do is actually talk to Google Assistant. Hey, Google– [CHIME] –turn on my washer. GOOGLE ASSISTANT: Sure,
turning the washer on. NETO MARIN: Oh, you
can see here the LED. OK, great. DANIEL MYERS: So
blue means it’s on. Now, if I start it,
it will actually start flashing various colors. Hey, Google– [CHIME] –start my washer. GOOGLE ASSISTANT: You got it. Starting the washer. NETO MARIN: So now it’s
like the washing cycle is running, right? DANIEL MYERS: Exactly. So that’s a various
cycle that you can start on a washing machine. Now, there’s also
other toggles– it’s another type of trait– that you can have
on a smart device. NETO MARIN: OK. DANIEL MYERS: Now,
if I turn that on– hey, Google– [CHIME] –turn on turbo for my washer. GOOGLE ASSISTANT: You got it. Turning on turbo on the washer. DANIEL MYERS: And so now
turbo’s on on the washer. And so that’s a various
toggle that you can control through voice as well. And ultimately, you can
turn it off as well. [CHIME] Turn off my washer. GOOGLE ASSISTANT: You got it. Turning off the washer. NETO MARIN: Cool, cool. So what’s happening here is,
like, Google Assistant is handling all the
things you said, calling an API at some
point, and sending the command to here, right? DANIEL MYERS: Exactly. NETO MARIN: OK, it’s a
very high level overview. Can you explain
some details, how this stuff is going back
and forth and working? DANIEL MYERS: Definitely. So at a high level,
you can think of it as cloud-to-cloud communication. NETO MARIN: OK. DANIEL MYERS: So
the first element is just me, the user,
saying a command– “turn on my washer.” Google Assistant
takes that audio, transcribes it into text, and
then applies natural language understanding, and ultimately
outputs a structured JSON response to your cloud service. So think of, like,
Philips Hue, for example. If you buy a Philips
Hue light bulb, that will have its own
application, its own cloud service, that you can use
to control those lights. NETO MARIN: And you have to
log into your own account, make all the process to assure
that you are not turning off your neighborhood lamp, so
something like that, right? DANIEL MYERS: Exactly. And so in this case,
Google Assistant communicates with their
cloud service, right? So if you’re turning on and
off a Philips Hue light bulb, Google Assistant communicates
with Philips Hue. NETO MARIN: OK, so
similar to that, we have for the
“regular action” what you call the webhook that can
manage the same high level architecture. So I have a webhook where
Google will call this webhook to send the commands, right? DANIEL MYERS: Exactly right. Exactly right. And so what I’ve
developed here today is my own cloud service that
accepts these structured JSON responses and then communicates
that information down to the device. NETO MARIN: OK. DANIEL MYERS: In this case, I’m
using Firebase as my transport down to the device. And this is looking at
various database states that it’s subscribed to. NETO MARIN: OK, cool. So basically, what
you need is, like, you need to set up your
webhook, your environment, your cloud servelets or
whatever you’re using, and enable these
to receive commands through JSON from the Google
Home, right, or the Google Assistant. And there is one word you
said, like, HomeGraph. DANIEL MYERS: Yeah. NETO MARIN: What is HomeGraph? What does it mean? DANIEL MYERS: Yeah,
so the HomeGraph is a very critical
element with Smart Home. It houses all of the contextual
information about you, your devices, and your home. So think of– you
might have five or six different smart light
bulbs in your house. You might have a smart washer. You might have a fan. You might have– NETO MARIN: Microwave. DANIEL MYERS: –microwave
or a thermometer somewhere– thermostat. All of that information is
stored in the HomeGraph. So when you say a command,
“turn on all the lights in my kitchen,” Google Assistant
is querying the HomeGraph to get a list of
all the lights that are actually in your kitchen. If you say, for example, “dim
the lights just a little bit,” it’s going to query and
see what the current state of the dimming is and then
dim it just a little bit. NETO MARIN: And
this is important. Because, for example, at my
house, I have me and my wife. So for example, I was the only
one that configured the lights and set it up on my Assistant. But she’s also able to
turn on and turn off. Because since she belongs to the
same home in my configuration, she has access to the
same HomeGraph, right? DANIEL MYERS: As long
as you share, you can– NETO MARIN: Of course,
you have to authorize– DANIEL MYERS: Right. NETO MARIN: Exactly. DANIEL MYERS: Exactly. And you know, it’s
important to note, too, this is a very
critical aspect, is that a smart home action
that uses the HomeGraph is distinctly different from
a conversational action. NETO MARIN: Yeah, this
is my next question. OK, great. DANIEL MYERS: And so with
a conversational action, you have to provide
various training phrases, and you can create
any custom intent that you want for your agent. NETO MARIN: You don’t
need to do this. DANIEL MYERS: That’s
exactly right. So for smart home,
we have a fixed set of intents, which include
things like onSync, onQuery, onExecute, and onDisconnect. NETO MARIN: OK. DANIEL MYERS: These are
the various commands that you can give
to a cloud service to do various
things with devices. NETO MARIN: One of the commands
you said was “turbo,” right? And this is because
you are washing– your washer supports turbo mode. If you try to change
to another program, I don’t need to train my action,
my home automation action, to know my different modes,
not because you understand the other phrase and
you send the parameters, then you handle these parameters
in your back end, right? DANIEL MYERS: That’s
exactly right. So that’s one of the beautiful
aspects of Smart Home, is that you, as the
developer, define the capabilities of the device. So if we take the example
of a smart light bulb, most smart light bulbs that I
know, you can turn on and off. Some you can dim. And others, you can
also change the color. NETO MARIN: The color, yeah. DANIEL MYERS: So these are three
distinct different capabilities of a light bulb. Now, how do you define that
so that Google Assistant can control those capabilities? We have a thing
called device traits. We have a huge collection
of different device traits that you can use to define the
capabilities of your device. NETO MARIN: Oh, OK. Nice, nice. So OK. You said developer,
as a developer. So as a developer, I’m excited. Want to build something
to my homebrew system. What are the requirements? What do I need to start
developing for home automation with Google Assistant? DANIEL MYERS: Yeah, so my
personal favorite route to go is our Smart Home Codelab
that we have online. NETO MARIN: Sure. DANIEL MYERS: The
biggest thing that you need is just a computer. You don’t even have to have a
physical device to get started. You can actually do it all
virtually through the Codelab. If you want, you can
definitely have a Raspberry Pi or any other type of device. NETO MARIN: These
boards are cool. DANIEL MYERS: Exactly. And my suggestion is to
take a look at the Codelab. The Codelab walks you
through every individual step of creating your first
smart device that you control through Assistant. NETO MARIN: OK, so cool. Let’s start coding
these and actually see how the parts fit together
and how the codes organize it. Yeah. Let’s show the code. People are shaking
sometime without code. Let’s see. DANIEL MYERS: Perfect. So the way I’ve
set this up today is using Firebase
Cloud functions. So I have here in my code
here various functions that I’ve deployed that we
use throughout the controlling aspect of this IoT device. NETO MARIN: OK, and you are
using the Node JS client library, right? DANIEL MYERS: Exactly. NETO MARIN: So we
already have a client lab that supports all the
interaction of the home automation, right? DANIEL MYERS: Yes. So we have a Actions on Google
client library for Node JS. You can see here, I also have
imported Firebase functions. And you can see
also that I’m using Firebase Admin for
my persistence layer to store the various
states of my device. NETO MARIN: OK, cool. DANIEL MYERS: And so
the biggest things that we need when developing
for Smart Home, as we stated earlier, are those
four fixed intents– the onSync, onQuery,
onExecute, and onDisconnect. NETO MARIN: OK. DANIEL MYERS: And so what
I have here is the onSync. What this does is
it provides a list of all the devices
for a particular user on your service. NETO MARIN: It’s
like Google querying for these titles of your stuff. DANIEL MYERS: Exactly. NETO MARIN: Your
connected stuff. DANIEL MYERS: Right. And so when you
initially go in– and say you just
bought a new light bulb and you want to control
it through Assistant, you have to go through this
initial linking process, account linking. NETO MARIN: Yeah. Listen, but when I’m on Home
hub, when I just swipe to left and I see all my
home stuff connected, like light bulb, thermostat,
it’s querying all my devices, calling onSync method for onSync
API for all my stuff connected and returning me the
[INAUDIBLE],, right? So something like that. DANIEL MYERS: Right. And so we see here that I have
one device called a washer. And we have a specific device
type of a washer as well. NETO MARIN: OK, quick question. So these types are defined
in our documentation, right? So you can’t just come up
with a [INAUDIBLE] device. I can’t come up
with a new device and just add a device there. It won’t work, right? Because it won’t
have the traits. Exactly. DANIEL MYERS: So you
can mix and match traits on different
device types. But you are correct that we have
a fixed set of device types. The reason why we have
these device types is that, for example, when
you actually say a command, we show an icon of a washing
machine, for example. We use the context of
the actual type of device in various ways
in the HomeGraph. NETO MARIN: Cool, cool, yeah. DANIEL MYERS: And so
you can see here I have five different traits or
five different capabilities of this device. So it can turn on and off. I can start and stop
the various cycles. I have a Run cycle. I have modes, and
I have toggles. And so those are the
capabilities of the device, and I can mix and match. If I don’t want it to be able
to have the Turbo toggle, for example, then
I can remove that, and you can no longer use Turbo. NETO MARIN: So some
features you don’t want to expose to
the home automation. You just don’t have the trait. DANIEL MYERS: Exactly. NETO MARIN: Yeah, like many
door lockers, they don’t have, for example, the unlock option. Because someone could
say, hey, unlock the door, and can open it and
go into your house. So you can hide some traits
if you think it’s critical, for example, right? DANIEL MYERS: Exactly. NETO MARIN: Cool. DANIEL MYERS: So you can
see here a little bit more about the attributes. So when we look at what modes,
so when you look at modes, you’re going to
have a set of modes. In our case, I’ve
set up two modes. There is a Small
Load and a Large Load for a washing machine. And so this is what it looks
like in our structured JSON response on how that’s defined
for giving it back to Google. NETO MARIN: OK, cool. So if, for example, I’m
defining a specific brand, and I have specific modes
that I want to expose. But then I launch a new
feature, I launch a new washer, then I can add new traits,
for example, or new modes. I can go here, update
my onSync method, and then don’t need
to deploy anything to the Google Assistant. It would be live,
available to test these new modes,
for example, right? DANIEL MYERS: Yeah. As long as you are developing
in a private state, you can continually update your
various traits, and your device types, and things like
that on your devices. And it’ll be able to
reflect on the device. You would still need to have
some type of on-device logic to be able to ingest
that information. NETO MARIN: Because at some
point, your, how can I say, server must communicate
to your stuff at home and send the command to you. So the washer must have
like a Wi-Fi board. And we must go
through the setup. If you’re a washer,
connect to your Wi-Fi and this kind of thing. So, yeah, I imagine you’re
talking about the home stuff, the Assistant stuff. But, yes, just keep in mind,
if you develop something new, you should still work
on the hardware side. We’re not giving
everything to you yet, but, yeah, you need to set up
the part of your own machine or your own device. It makes sense. DANIEL MYERS: And we
do that on purpose because we know that
developers want flexibility. We don’t define a
fixed approach on how your Cloud service communicates
and controls your devices. NETO MARIN: It makes sense. Cool. So for example, there
is a question I can say. The door is locked,
something like that, or just to see the status
when I open my Assistant and see this tells
of my stuff, it’s hitting this part, this method. In what other situations
this method is used? DANIEL MYERS: So the
sync, the actual response that’s returned, is also used
when you request to sync. NETO MARIN: Specifically,
I can request a sync. DANIEL MYERS: You can request
one from your Cloud service. So if a user already
has five lights and they go out and
purchase another light, they’re going to have
now a sixth light. If they want to
be able to control that sixth light in addition,
then, when they register it on your Cloud
service, it’s ideal that your Cloud service
then alerts Google, hey, they’ve added another device. Will you sync with me? NETO MARIN: Cool. And when you said, “Turn on
the washer,” how did it happen? DANIEL MYERS: So
what happens there is that actually goes through
onto the onExecute intent. NETO MARIN: Execute, OK. DANIEL MYERS: And so this
here is the execution intent of going through and looking
at the various inputs of the command itself. And so I look for
what exact command that it’s doing, so on/off,
start/stop, pause/unpause, set modes, or set toggles. NETO MARIN: And commands
were defined in your onSync, for example, when
you define the device or define what you’re
waiting for, right? DANIEL MYERS: Exactly. And so all of these you
can see match to the traits that we’ve defined
earlier, like you said. So the on/off matches– this is devices.commands.OnOff–
matches that of traits.OnOff. And so each different trait
has different commands that are associated with it. NETO MARIN: OK, cool. DANIEL MYERS: And so
another important aspect when looking at this
code is the onQuery. I can ask Google Assistant, is
my washer running right now? Another great question is,
if you have a refrigerator, you can actually
ask Google Assistant if your refrigerator is running,
and you better go catch it. NETO MARIN: It’s a
good question, yeah. DANIEL MYERS: And so
onQuery is the function that actually returns that state. NETO MARIN: OK, nice. DANIEL MYERS: And
so you can see here, it’s getting which devices the
user is asking about the state for. And then it’s able to get that
information and return it. NETO MARIN: Cool. DANIEL MYERS: Lastly, the final
and fourth intent that I want to go over is the onDisconnect. So for onDisconnect,
the disconnect intent is the one that shows when
a user unlinks their account with Google Assistant. NETO MARIN: Removed one of
these light bulbs, for example. DANIEL MYERS: Right,
they remove a light bulb. Or they no longer have
their washing machine. Maybe they sell it and
they no longer have it And so that’s what the
onDisconnect. is used for. And the second part that
I want to go over for this is the on-device
logic that I have. NETO MARIN: Yeah, cool, cool. Right now, for example, can you
go, for example, and execute code? I can see here that, in this
case, you are doing Firebase, and you are changing,
updating a field. And this is your logic. And then, I suppose
as an amateur maker, you have something
in your device that’s connected
to this database that you receive an
update that is valid, change it, and then can
do the hardware logic. Something like that? DANIEL MYERS: Right,
that’s correct. So you can see my
Firebase reference here. I have a specific field
of the deviceID that represents my washing machine. NETO MARIN: So you don’t want
to turn off your neighborhood washer machine. DANIEL MYERS: And then I
have a child called OnOff that I update the value of
to the parameter that was given to me in this request. NETO MARIN: And a
question– this was defined by you, for example, right? You are the hardware maker, you
are the owner of this process, and you define the structure. So our viewers are, OK,
but I don’t use Firebase. I use MQTT to send
messages from the function. No problem, right? DANIEL MYERS: No problem at all. NETO MARIN: Since it’s here
capturing this column from here and changing these toggles
on the actual machine, on the actual hardware, there
is no locking or dependence of having to use a Firebase
for this case, right? DANIEL MYERS: Correct. NETO MARIN: Even
though it can be like external API, the other API
on another server, the one you may isolate from
the home stuff, it can have another
call for API, right? DANIEL MYERS: Exactly. And so all of this,
you can abstract away to any type of persistence
layer, like you said. You can use anything
from GCP, for example, to store this information
and query later. NETO MARIN: Yeah, because
sometimes, for example, I just imagine– of course, use
Firebase, it’s a great tool. But maybe they already
have something implemented. They already have
an infrastructure for this hardware and
they want to implement it. So you don’t need to
migrate to Firebase first to integrate your hardware. So if you already
have a hardware and you are looking to integrate
Google Assistant to home automation, you can do it. You don’t need to use Firebase. All this is is a structure. So I think this is
the message, right? DANIEL MYERS: Right. Right. So when we look
at how the device ingests this database
information– let’s take a look at that. NETO MARIN: Sure. DANIEL MYERS: So this
is the file that I have. It runs on the Raspberry
Pi, on this AIY kit. And you can see
here a few things. One, I’m using Firebase
Admin as the way that I subscribe to
various elements. NETO MARIN: And look, this
is not Node.js, I’m seeing. DANIEL MYERS: Correct, yeah. This is Python. NETO MARIN: So I don’t need
even to use the same language. Because it’s not running on
the call function, right? DANIEL MYERS: Correct. NETO MARIN: So this code would
be running in the device? DANIEL MYERS: That’s right. NETO MARIN: Cool. DANIEL MYERS: So
this is everything that’s running on device. So you could have anything. You could be running Java. You could be running Node.js,
any number of other languages, and all doing the same
thing of listening for different database changes. NETO MARIN: So here you have the
logic for your washer machine. So this is the point here. And part of this
logic, of course, is read the state from the
home automation, right? DANIEL MYERS: Right. Right. So the things that
I’m controlling here is the color of the
light on the button of this washing machine and
then the various initial states that I set. It’s currently off, it’s not
on Turbo, it’s not running, and it’s currently paused. And so what we have here
is the various listeners and what I do inside
each of these listeners. I actually set up
the listeners here. So I’m referencing various
elements within my database. NETO MARIN: Cool, yeah, yeah. DANIEL MYERS: So a good
thing to take a look at here, actually, is let’s hop over
here and look at the structure of my database. NETO MARIN: Sure, cool. This is how you organized the
data for each device you have, right? This will reflect each washer. So I imagine this is the
database that the brand owns, right? And we’ll have like one registry
per linkage with the account, exactly? DANIEL MYERS: Right. NETO MARIN: More or less. Of course, they can
change the architecture. They can have different
authentication. But this register on the washer,
it’s specific for one hardware, right? DANIEL MYERS: Yes. So because this
is for a Codelab, there’s some hard
coding done here to just one washer and one user. So this kind of goes
as an MVP bare minimum of showing an example
of how this would work. NETO MARIN: Exactly. DANIEL MYERS: And so I’ve
expanded each element here to show what’s inside. And so when I have from here a
reference of /washer/OnOff/on/, what that’s actually looking
for is any changes to this field here. NETO MARIN: Let’s show this. Can you ask again to– not here. Say the command in your voice
on the phone to turn it on. Let’s see it changing
here in the database so they can see how it’s
connected to each one. DANIEL MYERS: And so
what you’ll see here is you’ll see all
of this change. You’ll see some colors change
here and the value of OnOff to become true. You’ll see this
also flip to be on. NETO MARIN: Yeah, quick
question– what is this? DANIEL MYERS: So this is– think of Philips Hue website
or the Philips Hue mobile phone app. This is the Codelab
front end for controlling your washing machine. NETO MARIN: So even
if you don’t have your device,
physical device, you can check the status
from this web page? DANIEL MYERS: That’s right. NETO MARIN: And even if
you’re in a testing mode, you can write this for your own
hardware and your own endpoint and check the
status through this. So you don’t need to
actually have the hardware all the time with you, right? DANIEL MYERS: Right, exactly. NETO MARIN: Because it’s
reading the same data, right? So cool. DANIEL MYERS: And so you’ll
see this change here. You’ll see this change here. And then you’ll also
see, in this terminal, the actual device
have an output of when it receives the command. NETO MARIN: OK, it’s like your
debug window for the device. DANIEL MYERS: Right. So I’m SSHed into my
Raspberry Pi here. NETO MARIN: Cool. DANIEL MYERS: So
let’s watch it happen. NETO MARIN: Do the show. Turn on my washer. GOOGLE ASSISTANT: OK,
turning on the washer. NETO MARIN: Cool. I see there. It’s true here. Toggle there is on. DANIEL MYERS: And it showed
that it’s turning on. So all three elements
there in near real time is event-based where,
based on your voice, it’s doing all of these things. NETO MARIN: So it’s good. Because you said the command,
the system parsed the command and saw there was
something to your washer, checking your HomeGraph, say,
yeah, he has a washer, then called this webhook
with the onExecute, sent the trait, the driver. You updated the database. And it was like immediately. DANIEL MYERS: Right, that’s the
beauty of all of this going. It’s all event-based. It happens so fast. And that’s why I like doing IoT. NETO MARIN: Yes, man, it’s cool. Personally, it’s
something I really like. I’m able to spend a weekend
connecting wires and trying to run random stuff. It’s cool. So cool. I think we’ve covered
a lot of stuff here. Can you just recap
some steps that they have to keep in mind
when developing this? DANIEL MYERS: For sure. So from a high level,
the big thing to remember is that this is cloud-to-cloud
communication where the developer responsibility
is taking in a structured JSON response from Google,
applying some logic to that of what state the
device should do, and then actually
controlling the device. And so from our perspective
here, the way we’ve set this up is that we’re using
Firebase Cloud functions for the
various intents that we’re listening for. And then we’re using Firebase
as the persistence layer to store the device state. And then the Python
application that’s running on the
Raspberry Pi listens for those various states. NETO MARIN: Cool, yeah. This is so cool. I’m very excited because
I like this stuff. I like IoT stuff. So, yes, thanks for being here. Thanks for sharing. This is amazing. I think if they want
to find us, they can reach us through our
team’s Twitter handle, @ActionsOnGoogle. And if you’re sharing
something with us, also use the hashtag #AoGDevs
so we can see your cool stuff or making question to us. It would be great. And also, we have the
Reddit Actions on Google, so you can ask questions,
meet other people from the community. And if they have a
question, where they can go for technical questions? DANIEL MYERS: Stack Overflow. NETO MARIN: Stack Overflow. So Actions on Google
would be great. And thanks for watching. Please give your comment,
your feedback, what you think, what next episodes
you are waiting for. And we’ll be glad
to produce for you. OK, see you next time. DANIEL MYERS: Thanks [JINGLE]

17 thoughts on “Programming your home automation with the Google Assistant (Behind the Actions, Ep. 3)”

  1. Make it possible to disable confirmations! It's so annoying after a while! "Okay, turning of the lights!"

  2. Please give us a LOCAL WEBHOOK FEATURE!
    Add to the Google Home App a "local webhook first" mode, so I can use my local (and therefor secure) server for home automation (and much more):
    Just let me define in the App an URL (in my local network) to handle ALL requests, e.g. via input param q=[spoken request as text].
    This hook (after doing some work behind the scenes) just returns a string reponse, which will be spoken by my Google Assistent device.
    When this response is empty, Google Assistent handles the request as usual (i.e. with the help of the Google services in the internet).

    More secure (no internet involved, maybe apart from language processing), more reliable (still no internet, so no extern clouds or devices involved), super flexible (the limits are only your own programmung skills), and much more faster!

Leave a Reply

Your email address will not be published. Required fields are marked *