Google Developers community, our Medium has had some compatibility errors. Like an outdated SDK, our Medium account no longer has proper support nor up to date documentation. To avoid reduced developer performance, it’s time to sunset it.

But we’re not going to leave you stranded, like a 404 error message. Instead, stay connected by following us on our channels:

• Twitter
• LinkedIn
• Instagram
• Blog

Here, we’ll continue to provide you with the latest tech trends, tutorials, and developer insights across mobile, web, AI, and Cloud. Think of it as upgrading from an old version of an SDK to a new and improved one!

So, to all you developers who have read, clapped, and commented on our Medium posts, we want to say thank you for your support.

Sincerely,
The Google for Developers Team

Error 404: Google Developers Medium account not found. was originally published in Google Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

Break out from the sea of Android App icons — What are Custom Intents for App Actions?

I counted and I have 129 apps installed on my phone, not including my work profile! I doubt that I’m an outlier.

Do I use all of them? Nope, I totally forgot that I had installed so many of them. The biggest challenge for android developers after installation is usage rate, which in 2016 it was reported that there is 77% Drop off rate of usage 3 day post install (ref).

But there are a few apps that I rarely “open” but I use on a regular basis. I don’t mess with the sea of icons, even though I have the app on my home screen, I normally use my voice to open it. Its like magic.

Almost every morning between 6am and 7am, I say “Hey Google, open fit and start walking” and it’ll open my Google Fit app and start tracking my walking. As a user I don’t even have to think about finding it in the sea of icons or take the extra step to add it to my home screen. It’s available when I call for it — truly meeting me where I am.

You can do this with App Actions, it’s a simple way to allow Android developers to use the Assistant to “fast-forward” users to exactly the right spot in their app with a voice request. You might have heard of BIIs (built-in intents), which models some of the common ways that users express tasks they’re trying to do or information they seek, from your Android Apps. The most common BII that all Android Apps can utilize is Open App Feature, which allows users to say “Hey Google, open <app name>” and it’ll open the app. Opening the app is great, but going to the right place in the app is magical. The ability to say “Hey Google, open <app name> AND do x” can be achieved with BIIs.

But Android apps are not all the same. They are unique and provide particular value to their users. For the functionality that isn’t supported with BII, Custom Intents gives you complete flexibility to allow your users access to your unique functionality. deep links. There are some key differences between BIIs & Custom Intents.

Biggest differences between BIIs & Custom Intents

Functionality

When it comes to the type of functionality that can be surfaced by your App, BIIs have a fixed number. Though this fixed number of BIIs are the most popular type of functionality like opening the app, getting a thing, create a thing and more which are organized by categories, such as communications, games, shopping and social. But what if I have a “Jessica’s Baking App” that provides Jessica’s super special recipes? There isn’t a baking BII. *trumpet sounds “ba ba-da ba-ba bah”* Custom Intents lets me handle my user’s request to start baking a strawberry cake!

Queries

In order for the Assistant to understand how users can trigger your custom intent, you’ll need to provide query patterns for the phrases it should recognize. Query Patterns are a set of strings that are different ways a user can express their intent. Each query pattern contains the words after “Hey Google, open <App Name> and …” (example “Hey Google, Open Jessica’s Baking App and start baking a cake”)

Since the Assistant matches users’ requests based on these query patterns, having a variety of ways a user can express this intent is ideal. People are complex and come from different places (geographically, emotionally, culturally, etc.) and use different words to convey the same messing. Capturing the different ways is super important to make sure that the Custom Intent gets triggered. I recommend reaching out to different people and asking them how they would express the desire for that functionality so your query patterns are as robust as possible.

For my app, called Jessica’s Baking App, I know that I would say something like “make” or “bake”, but someone else might say “craft” or “produce” instead. This part is super easy to overlook, but spending the time and energy to gather a nice list of query patterns will pay off in the end since the Assistant doesn’t do any magic for you beyond the query patterns you specify. In other words, if I only have“start baking a cake” in my query patterns, the custom intent won’t trigger for when someone says “let’s cook a cake” or “build a cake” even though they are similar. You’ll have to be explicit and thorough.

For BIIs, there is no need for query patterns since they are already pre-defined. These pre-defined query patterns have been curated by language design experts and are very extensive.

Locales | localization

Since BIIs query patterns are maintained by Google, they have support for a variety of locales. Some of them have several, the newer ones may only be en-US. Check out the docs to see which languages are supported for a particular feature.

At the moment, Custom Intents are only available for the en-US locale.

Parameters!

When using BIIs or Custom Intents, you can capture sections of what the user said as parameters. These parameters can be used to display the right content in your App or to auto populate content. With Custom Intents, you have to define your parameters, unlike BIIs which are predefined. You’ll need to define parameters within the Custom Intent element and in your query patterns.

For example, in my example Jessica’s Baking App if a user says “start baking a cake” I can capture cake as a bakingType parameter and provide a generic cake recipe. If a user says “produce an apple pie” then I can provide an apple pie recipe instead.

Being able to give your users a faster way to open your Android App and go to the relevant screen via the Google Assistant can help with breaking through the sea of icons. It also makes a multi-step process much shorter — no need to browse through a list of apps, then a list of things you might bake, then hit go… you’re right there right away. Better productivity! You can do this with App Actions by either implementing BIIs or Custom Intents.

Tips & Resources

Interested in implementing Custom Intents? Below are a few tips and resources to help you out:

• Check out the list of BIIs to see if the functionality you want is already supported and use that! If there isn’t one, then go ahead with Custom Intents
• Crowdsource a variety of ways a user can invoke the Custom Intents for your query patterns
• Check out the docs on how to implement
• Test out your App Action with the App Action test tool

Thanks for reading! To share your thoughts or questions, join us on Reddit at r/GoogleAssistantDev.

Follow @ActionsOnGoogle on Twitter for more of our team’s updates, and tweet using #AoGDevs to share what you’re working on. Can’t wait to see what you build!

Break out from the sea of Android App icons — What are Custom Intents for App Actions? was originally published in Google Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Google Smart Home platform lets users control your connected devices through the Google Home app and Google Assistant, helping to create a helpful home for your users. As the Smart Home platform has continued to mature over the past year, we’ve created additional tools and resources to help you develop your own Smart Home Action — check them out below!

Enhanced Analytics and Logging

To help you monitor and log events within your Smart Home Action, Enhanced analytics and logging tools launched on June 30th. These pre-populated metrics charts are accessed directly through the Actions console, and leverage Google Cloud Monitoring features. Track device traits most commonly accessed by your users, latency for user query responses, success rates of user engagement, and compare interactions between cloud and local fulfillment. Additionally, log events provided through Google Cloud Logging help you debug your Smart Home Action quickly by accessing event logs that include information on the intent type, device type, fulfillment pathway, and the locale of the request.

These features will help you quickly identify issues within your Smart Home Action, and provide quality information to help guide continued development and improvements to your end-user experience.

Enhanced Analytics and Logging Videos:

• How to use error logs
• Enhanced Smart home Analytics

Local Home SDK

The Local Home SDK exited developer preview on April 6th, and version 1.0 was made publically available to all developers. Based on feedback from the developer preview, we added the ability to enter multiple scan configurations to the Actions console. Also the console now supports uploading Javascript bundles for your local fulfillment app.

Since the launch of 1.0, we’ve also added some additional features including Node.js runtime support to the Google Nest WiFi routers. This means the Local Home SDK is now compatible with the full Nest WiFi system. The Node.js on-device testing URL can be added to the console, also enabling you to use Chrome DevTools for inspecting and debugging Node.js apps. Other small fixes have also been added — you can find more information in the source code here.

By implementing the Local Home SDK within your fulfillment app, your Action can be more reliable during fulfillment of user requests by routing commands over the local area network, while still maintaining the cloud fulfillment path as backup.

Local Home SDK Samples:

• Smart Home Local

Local Home SDK Codelabs:

• Enable local fulfillment for smart home Actions

Smart Home Entertainment Devices (SHED)

Smart Home Entertainment Devices (SHED) types and traits — To support partners with entertainment focused devices, we launched multiple new entertainment device types, and media-related traits in April and July. These enable full voice control and engagement with entertainment devices within the home from any Assistant-enabled device. To see the complete list of supported traits and types, check out the reference documentation.

App Flip

To help ease the account-linking process for users to onboard with your Action, App Flip launched in July. This easy-to-develop feature simplifies the process for account linking via mobile, by enabling users to easily link their partner accounts to Google Assistant without having to manually log in again.

App Flip helps enhance your existing OAuth 2.0 server implementation, and you can easily utilize this feature within your Android or iOS app with a minimal amount of code updates.

Scheduled Smart Home Actions

Scheduled Smart Home Actions launched in November, which enables users to schedule device commands to trigger at a later time . This feature is supported for all Smart Home Actions, and does not require any additional development work. Currently, the traits `LightEffects`, `OnOff`, and `StartStop` support scheduling.

Light Effects Emulation

Another user-facing feature released in September was Gentle Wake/Sleep light effects. This enhancement enables users to slowly brighten or dim their lights, switches, and plugs over a preset time period. These device features are supported by default through the `LightEffects` trait. If your Smart Home Action currently only uses the `Brightness` trait, your end-users can still benefit from Gentle Wake/Sleep thanks to emulated light effects support. This feature also does not require any development work from your teams to be compatible with your Action.

General Platform Resources

Finally, we introduced several general resources to help you with your Smart Home Action development. In an effort to open Smart Home Action development to all programming languages, we released the Smart Home Schemas repository. This contains the JSON schemas that define the Google Smart Home platform. We have updated our documentation to reflect these schemas, ensuring better accuracy with full example intents in the device guides, and more detailed payload parameters within the traits’ references.

New Platform Videos:

• How to self-certify Smart Home Actions for the Google Assistant
• Account linking verification for smart home
• Inspecting Home Graph
• Test suite for smart home
• Offline handling for smart home

New Platform Blogs:

• Account Linking Verification for Smart Home
• Inspecting Home Graph
• Test Suite for Smart Home
• Handing Offline Devices

Great Learning Resources:

• Docs: goo.gle/assistant-smarthome-docs
• Samples: Samples and libraries | Actions on Google Smart Home
• Codelabs: All Smart Home Codelabs
• Playlist: Smart Home video Playlist

More Resources:

• Ask your technical questions on Stack Overflow
• Stay up-to-date with announcements by following us on Twitter
• Connect with other Assistant Developers on our Reddit community

Interested in Conversational Actions or App Actions? Check out their own blog with all new features and resources listed out at:

• Conversational Actions 2020 Updates & Resources
• App Actions 2020 Updates & Resources

Thanks for reading! To share your thoughts or questions, join us on Reddit at r/GoogleAssistantDev.

Follow @ActionsOnGoogle on Twitter for more of our team’s updates, and tweet using #AoGDevs to share what you’re working on. Can’t wait to see what you build!

Smart Home Actions 2020 Update & Resources was originally published in Google Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

• Actions Builder & Actions SDK
• Interactive Canvas
• Discovery

Build with Actions Builder & Actions SDK

Actions Builder and Actions SDK was launched in June, making it easier to design and build conversational Actions on Assistant, like games, for Smart Displays. Actions Builder gives you a powerful and easy-to-use, web-based IDE to develop, test, and deploy in the Actions console. The graphical interface lets you visualize the conversational flow, manage Natural Language Understanding (NLU) training data, and debug with advanced tools.

The updated Actions SDK gives you a file based representation of your Actions project. You can build your conversation and do bulk imports and exports of training data with the updated command-line interface (CLI).

Since Actions Builder and Actions SDK are built on the same technologies, you can develop Actions using your preferred tooling; allowing teams to collaborate on the same project.

Improvements since launch

We have continued to improve Actions Builder and Actions SDK with version management, testing and analytics to help you better manage your Action development. The Actions SDK now lets you list versions and specify which version to download, allowing you to configure an Actions project to the local file system. If you’re developing using VC Code, check out the VS Code extension for Actions SDK for features like autocompletion and error checking on all Actions SDK YAML files. The Actions SDK now supports testing with the Node.js Testing Library, so you can build automated tests to make sure that your Action is responding as expected. After you have an alpha, beta or production version of your Action published, in the Actions Console you can now see analytics for each type of release. This is ideal for evaluating your alpha and beta Actions before pushing to production.

With the popularity of smart displays and that they are usually located in communal areas, support to be helpful to everyone in the household has been added with Home storage. Home storage allows you, the developer, to store values over multiple sessions in the same household (based on Home Graph). This feature is ideal for games that engage the household.

Migrate to Actions Builder & Actions SDK:

If you have an Action built using Dialogflow, you can migrate your project to Actions Builder within the Actions Console. The Dialogflow to Actions Builder migration tool handles the migration of your Dialgoflow agent to an Actions Builder project. Then you can Fulfillment Migration guide to update your fulfillment code. For an overview check out Migrating from Dialogflow to Actions Builder video.

Below is a list of resources of code samples, codelabs and videos about Actions Builder & Actions SDK:

Actions Builder & Actions SDK Code Samples:

• Hello World sample
• Conversation Components, includes automated tests
• Transaction
• Action Builder Patterns
• Custom NLU
• Account Linking
• Localization
• Facts about Google

Actions Builder & Actions SDK Codelabs:

• Build Actions using Actions Builder Level 1, Level 2
• Build Actions using Actions SDK Level 1, Level 2

Actions Builder & Actions SDK Videos:

• Overview of Actions Builder & Actions SDK
• Actions Project overview
• Intents overview
• Scenes overview
• Creating an Action with Actions Builder
• Creating an Action Actions SDK

Interactive Canvas

With the launch of Actions Builder and Actions SDK, the Interactive Canvas API has been updated and improved upon. There are more ways of sending data between your Actions Project, fulfillment and Interactive Canvas web app. You can send data from your Interactive Canvas web app to your fulfillment with setCanvasState(), and you can send data from your Actions project to your Interactive Canvas web app without needing to go through your fulfillment implementation. To help minimize the repetition of adding the url to your Canvas responses, you can now add your default web app url in the Actions Console.

Improvements since Action Builder & Actions SDK launch

Debugging your Interactive Canvas Action has gotten easier by using Chrome DevTools for Google Home or Google Nest smart displays with a software version of 30 or later. Or in the simulator, you now get feedback from client side errors.

We launched several games that use Continuous Match Mode which lets you leave a device’s microphone open for a set period of time while listening for a list of words you specify.

The Games Portal was launched in September with tons of resources to build great games on Assistant. You’ll find things like our new game design guide, code samples, interviews with the game designers and developers and case studies.

Below is a list of resources of code samples, codelabs and videos about Interactive Canvas:

New Interactive Canvas Code Samples

• Snowpal Canvas sample is a variation of the traditional Hangman game; the game presents you with a number of blank spaces representing a word, and you guess letters that you think may be in the word.
• Cookie Detective is a game for kids to become kitchen sleuths as they locate their cookie friend under 10 guesses. The game is implemented using Interactive Canvas for Nest Hub and Nest Hub Max smart displays
• Gnome Garden lets you sculpt a zen garden with your voice. This game has been designed for Nest Hub and Nest Hub Max smart displays and implemented using Interactive Canvas

New Interactive Canvas Codelab

• Build an Interactive Canvas Action with Actions Builder

New Interactive Canvas Blog

• Synchronize animations with the Text-To-Speech

New Interactive Canvas Videos

• Building games on smart displays
• Game design process
• Game development and tools
• Continuous Match Mode party games
• Game personas and visuals
• Opportunities and next steps

Discovery

In October Actions links were renamed to Assistant Links. Assistant Links are URLs that link directly to a specific intent of your Action; which are great for email marketing materials. Assistant Links have been expanded with Rich Assistant Links, which allows you to access your Action’s discovery through your website by using assist.js.

Great Learning Resources:

• Docs: goo.gle/assistant-conversational-docs
• Samples: Conversational Actions Samples
• Codelabs: Conversational Actions Codelabs
• Videos: Conversational Action video playlist

More Resources:

• Ask your technical questions on Stack Overflow
• Stay up-to-date with announcements by following us on Twitter
• Connect with other Assistant Developers on our Reddit community

Interested in App Actions or Smart Home Actions? Check out their own blog with all new features and resources listed out at:

• App Actions 2020 Recap Blog
• Smart Home 2020 Recap Blog

Thanks for reading! To share your thoughts or questions, join us on Reddit at r/GoogleAssistantDev.

Follow @ActionsOnGoogle on Twitter for more of our team’s updates, and tweet using #AoGDevs to share what you’re working on. Can’t wait to see what you build!

Conversational Actions 2020 Recap was originally published in Google Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

In 2020, we launched 30+ more Built-in Intents, bringing the overall number to over 60. New common Built-in intents, such as “Open app feature” that opens specific activities in the app, or “Get thing” that searches for content using the default in-app search feature, are usable for the majority of apps. In case your app category isn’t listed, you can use Custom Intents. Custom intents, you need to provide the query patterns, which are different ways a user might say to express this activity.

For Built-in Intents such as order menu item or get exercise plan, the fulfillment URLs can leverage web inventory. In other words, Assistant is using a google search index of your public web content to forward a URL to the app that’s responsible for rendering it in a native Android experience. Web inventory uses your website to discover URLs for App Action fulfillment. This feature is most useful when you have a strong web presence, and your in-app deep links are organized around publicly available web content.

Foreground app invocation allows built-in intents to be matched without requiring the app name to be mentioned while a specific activity is in the device foreground. For example, if a user has your rideshare app in the foreground and says or types, “Order me a ride to Mountain View.” to Google Assistant. Your app uses this input to set the destination field to Mountain View. Then, when the user says or types, “Order me a ride from SFO,” your app can additionally set the pickup location while preserving the state of your app.

To further support developers, the App Actions test tool was launched. It provides a mechanism in Android Studio to test app actions prior to submitting a new version of the app in the Play store.

Below is a list of resources of code samples and videos that were launched this year:

New Samples:

• App Actions to-do list sample Users can add items to to-do lists, search for items by category, and view information about completed tasks.
• Get analytics for App Actions using Google Analytics Sample log events using the Google Analytics for Firebase SDK. You can then use Firebase Console to visualize traffic patterns and obtain app usage and user engagement insights.

New Videos:

• Extend your Android application with Google Assistant
• A Conversation about Android app

Great Learning Resources:

• Docs: goo.gle/assistant-app-docs
• Samples: App Actions Samples
• Codelab: Extend an Android app to Google Assistant with App Actions
• Playlist: App Action Video Playlist

More Resources:

• Ask your technical questions on Stack Overflow
• Stay up-to-date with announcements by following us on Twitter
• Connect with other Assistant Developers on our Reddit community
• Check out our All Assistant videos playlist

Interested in Conversational Actions or Smart Home Actions? Check out their own blog with all new features and resources listed out at:

• Conversational Actions 2020 Recap Blog
• Smart Home 2020 Recap Blog

Thanks for reading! To share your thoughts or questions, join us on Reddit at r/GoogleAssistantDev.

Follow @ActionsOnGoogle on Twitter for more of our team’s updates, and tweet using #AoGDevs to share what you’re working on. Can’t wait to see what you build!

App Actions 2020 Recap was originally published in Google Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

Missed our weekly video? Don’t worry, watch this week’s #AoGProTips 🎥

Many factors can affect device connectivity in the home, and it’s important to keep users aware when their devices are unreachable or offline. A critical element of your smart home Action is properly maintaining and reporting the reachability of each device to Google Assistant.

The smart home API supports multiple ways to indicate that a device is unreachable, so let’s explore when it’s appropriate to use each one.

Offline status

Each response that your smart home Action provides to a QUERY and EXECUTE intent should include a status code. For most intents, the status value will be SUCCESS. When Google Assistant attempts to QUERY an unreachable device, you should return the device status as OFFLINE. This indicates that there is no additional state available from the device at this time.

{
  "requestId": "ff36a3cc-ec34-11e6-b1a0-64510650abcf",
  "payload": {
    "devices": {
      "123": {
        "status": "OFFLINE"
      }
    }
  }
}

Offline error

Each EXECUTE intent represents a command to change the state of the device. If that command fails to execute for any reason, it is appropriate to report that condition with an error response. When an EXECUTE intent fails because the device was unreachable, return an ERROR status with the error code set to deviceOffline.

{
  "requestId": "ff36a3cc-ec34-11e6-b1a0-64510650abcf",
  "payload": {
    "commands": [
      {
        "ids": [
          "123"
        ],
        "status": "ERROR",
        "errorCode": "deviceOffline"
      }
    ]
  }
}

Note that if your integration handles commands asynchronously or any other conditions require you to return a PENDING status in the EXECUTE response, you can update the status of an unreachable device using the offline state flag.

Offline status flag

Finally, we have the online state flag. Use this flag to publish any changes in device connectivity through the Report State API. This ensures that Home Graph remains up to date and reduces the chance that the Assistant sends QUERY or EXECUTE intents to an unreachable device.

{
  "requestId": "ff36a3cc-ec34-11e6-b1a0-64510650abcf",
  "agentUserId": "1234",
  "payload": {
    "devices": {
      "states": {
        "123": {
          "online": false
        }
      }
    }
  }
}

For more helpful tips on getting the most out of your actions, be sure to check out the rest of the AoG ProTips series — and share your tips with us on Twitter using the hashtag #AoGProTips.

AoG ProTips: Handing Offline Devices was originally published in Google Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

Missed our weekly video? Don’t worry, watch this week’s #AoGProTips 🎥

The test suite for smart home allows you to self-test your smart home Action and verify that it meets all the necessary criteria before you submit for review. The test suite automatically generates test cases for the supported devices and traits reported by your Action. The test suite cannot verify traits which do not have state (such as Locator) or cannot accept commands (such as RunCycle).

In this post, we’ll explore some of the best practices to ensure that your test suite runs are successful!

Can you hear me now?

The test suite generates text to speech (TTS) audio commands for each device trait reported by your Action. The commands must be heard by a nearby Google Home or Nest device in order to reach your smart home Action. If the device hears the command incorrectly, it will not execute the correct command and the test will fail.

Note: This also means that you must ensure the Nest device you are testing with is linked to the same user account that you supplied to the test suite.

Ensure that you run the test suite in an environment where the Nest device can hear your computer speakers clearly without background noise. You might even consider putting headphones over the device microphones to block out any external sound!

Implement Report State

The test suite verifies command results by inspecting the state in Home Graph for the expected value. The test suite does not verify the EXECUTE response or send a QUERY intent to determine success, so you should test those portions of your smart home Action separately. If you do not update Home Graph directly after each command, the tests will fail.

To update Home Graph, you must implement the Report State API in your smart home Action to notify Google of asynchronous changes with your devices, and enable the willReportState flag for each device in your SYNC response — the test suite will skip devices with this flag set to false.

{
  "requestId": "ff36a3cc-ec34-11e6-b1a0-64510650abcf",
  "payload": {
    "agentUserId": "1836.15267389",
    "devices": [
      {
        "id": "123",
        "willReportState": true
        ...
      }
    ],
  },
}

Report state frequently

It’s critical for your smart home Action to maintain accurate data in Home Graph at all times. When Google Assistant does not find recent state data in Home Graph, it must issue a QUERY intent to your service to determine the current state, adding latency to the user experience. You should also call Report State after any command sent to the device, even if the command does not trigger a state change (e.g. the light is already on).

Following these steps ensures Home Graph is accurate starting from the initial SYNC, and avoids common failure cases in the test suite due to missing or incorrect Home Graph data for the devices under test.

Submit your results

After completing your test run, it’s time to submit your results through the certification form in the Actions console. During submission, you will have the opportunity to note any device traits that require additional testing and add comments to your results to indicate if any failing tests are due to design considerations in your devices.

For more helpful tips on getting the most out of your actions, be sure to check out the rest of the AoG ProTips series — and share your tips with us on Twitter using the hashtag #AoGProTips.

AoG ProTips: Test Suite for Smart Home was originally published in Google Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

Missed our weekly video? Don’t worry, watch this week of #AoGProTips 🎥

When building smart display games for the Google Assistant using Interactive Canvas, you can add fun animations to create a fully immersive experience. Do you know you can synchronize your animations with audio, such as making a dinosaur open its mouth at the exact moment you play the roaring sound? In this blog post, you will learn how to use the SSML mark tag with the onTtsMark callback function to synchronize your animations with audio.

SSML stands for Speech Synthesis Markup Language. By using SSML, you can make your conversation’s responses sound more natural by adding breaks between words, and adjusting the speed, pitch and rate of a word. Look at the SSML example below.

<speak> The dinosaur is about to roar <mark name = ‘START_ROAR’><audio src=’roar.mp3’/><mark name =’STOP_ROAR’></speak>

In the SSML, the <mark> tag allows you to indicate during the generated TTS audio when the dinosaur should start and stop animating. It generates events during TTS; your code has a callback that gets triggered by each mark tag. Each mark event has a name. In this example, we have two events named “START_ROAR” and “STOP_ROAR”. You need to write code that can be triggered by each mark event. The code for the “START_ROAR” event can open the dinosaur’s mouth after the spoken prompt “The dinosaur is about to roar “ is complete. Similarly, the code for the “STOP_ROAR” event can close the dinosaur’s mouth.

Now that the mark tags are in place within the SSML, you can write the logic for each animation when the mark tag is hit. Keep in mind, the name of the mark tag must be unique within the SSML so that the onTtsMark callback responds to the correct cue. Let’s look at the code below.

You register a callback for onTtsMark. The onTtsMark() method receives the markName in the SSML, and depending on the name of the mark tag, triggers the corresponding logic which plays the animation of the dinosaur moving its mouth. For the ‘START_ROAR’ markName, it will call the beginRoaring function, whereas ‘STOP_ROARING’ will call the stopRoaring function to stop the animation.

Now that you have learned how to synchronize animations using the SSML mark tag and the onTtsMark callback, we hope you apply this tip in your next Action to create a fully immersive gaming experience for your users.

If you have a tip that you think other developers should know about, share your thoughts with us on Twitter using #AoGProTips. Lastly, check out our collection of other pro tips here.

AoGProTips: Synchronize animations with the Text-To-Speech was originally published in Google Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

Google’s Home Graph provides contextual data about the home and its devices, creating a logical map of the home. This enables users to target devices by room or implicitly make commands when you are located in the same room. Most of the data is populated by your smart home Action, and accurate Home Graph data is critical to providing the best user experience through the Assistant.

In addition to the Report State and Request Sync methods that your cloud service invokes, the Home Graph API includes methods that are useful during development to validate the device capabilities and state data reported by your Action.

Let’s explore how we can use the Home Graph API to validate the SYNC response and debug Report State.

Obtain a service account key

The Home Graph API requires authorization to access the device data in your project. Use the following steps to create a new service account credential associated with your smart home Action project.

1. In the Google Cloud Platform Console, go to the Create service account key page.
2. From the Service account list, select New service account.
3. In the Service account name field, enter a name.
4. From the Role list, select Service Accounts > Service Account Token Creator.
5. For the Key type, select the JSON option.
6. Click Create.

Download the JSON credentials to your computer as service-account.json.

Initialize a Home Graph client

The Google APIs Node.js Client library includes bindings for the Home Graph API, so we can build a simple Node.js script as a client interface with just a few lines of code. You can install the client library as a dependency using NPM:

$ npm install googleapis

Inside your script, use the following code to instantiate an authenticated client for the Home Graph API with your JSON service account credentials.

const {google} = require('googleapis');
const credentials = require('./service-account.json');

// Create an authorized client for Home Graph
const auth = new google.auth.GoogleAuth({
  credentials: credentials,
  scopes: ['https://www.googleapis.com/auth/homegraph']
});

const homegraph = google.homegraph({
  version: 'v1',
  auth: auth,
});

Note: Home Graph API bindings are also available in Java.

Inspect SYNC metadata

Using the sync method, you can verify the metadata in Home Graph for the devices associated with a given agentUserId provided by your smart home Action.

// Request set of devices for the given user
async function sync(homegraph, userId) {
  const request = {
    requestBody: {
      agentUserId: userId,
    }
  };

  return await homegraph.devices.sync(request);
}

This data is populated in Home Graph by SYNC intent response, and it’s a good way to verify that the types, traits, and attributes for each of your user’s devices are reported properly.

Query device state

Using the query method, you can request the state data stored in Home Graph for devices associated with your smart home Action. This data is provided by your Report State implementation, keeping Home Graph state in sync with your service. You should verify that the data for each device is not missing or incomplete.

// Request the current state of a given user's device
async function query(homegraph, userId, deviceId) {
  const request = {
    requestBody: {
      agentUserId: userId,
      inputs: [{
        payload: {
          devices: [{
            id: deviceId
          }]
        }
      }]
    }
  };

  return await homegraph.devices.query(request);
}

Note that QUERY intent responses do not update Home Graph state. If you see incomplete state data in Home Graph, it means you need to modify how and when you call Report State to provide a more complete picture.

Report State Dashboard

An alternative way to visualize your Home Graph data is with the Report State Dashboard. The dashboard is a frontend web application written in Java that provides a graphical interface to review device states for a given agentUserId in your project.

Download the source from GitHub and follow the instructions in the README to get started.

For more helpful tips on getting the most out of your actions, be sure to check out the rest of the AoG ProTips series — and share your tips with us on Twitter using the hashtag #AoGProTips.

AoG ProTips: Inspecting Home Graph was originally published in Google Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

Smart home Actions rely on account linking to authorize the Google Assistant to access a user’s devices, but this is a multi-step process that can be difficult to debug in production. When you encounter an OPEN_AUTH_FAILURE in your StackDriver logs, you need a way to interactively debug each step of the account linking integration to track down the root cause of the issue.

Using Google’s OAuth Playground tool, you can interactively step through the account linking process with your cloud service in a secure way and verify whether Google is able to obtain the necessary authorization tokens. Let’s explore how to configure and use OAuth Playground.

Configure OAuth Playground

Navigate to your account linking configuration for your project in the Actions console. You will use this information to configure the OAuth Playground tool.

Launch OAuth Playground, and click on the gear icon in to open the OAuth 2.0 configuration dialog. Configure the interface as shown using the OAuth Client information from your Actions console project:

• Authorization endpoint: Set this parameter to the Authorization URL in the console.
• Token endpoint: Set this parameter to the Token URL in the console.
• OAuth client ID: Set this parameter to the same value as in the console.
• OAuth client secret: Set this parameter to the same value as in the console.

Link an account

We can now walk through the steps to obtain an access token from the OAuth endpoint.

Select & authorize APIs.

Enter any additional scopes your service requires in the text field that says Input your own scopes. If your service does not require any, enter something generic like “devices” and click Authorize APIs.

This will redirect to your Authorization URL to sign in. Once the sign in flow successfully completes, you will be redirected back to the OAuth Playground for step 2.

Exchange authorization code for tokens

This section is pre-filled with the authorization code returned by your service. Click Exchange authorization code for tokens to request token credentials from your Token URL.

If successful, the refresh token and access token will populate in the UI.

Verify that you can successfully execute a token refresh against your account linking endpoint by clicking the Refresh access token button in the Playground UI. If you get an error at this stage, it is likely because your service is not properly accepting the refresh token it initially generated as valid.

Check for errors

OAuth Playground clearly displays the request and response for each step, allowing you to determine exactly which stage in the account linking process may be failing.

If you do not encounter any errors obtaining a user access token from your service, you can attempt to use that token to invoke a smart home intent.

Send an authorized request

The final step is to verify your service will successfully accept the credentials attached to a smart home intent request. Let’s configure the tool to send a sample SYNC request.

Configure request to API

Expand this section in the Playground UI, enter your fulfillment URL from the Actions console as the Request URI, and set the HTTP Method to POST. Verify that Content-Type is set to application/json and enter the following sample request body:

{
  "requestId": "ff36a3cc-ec34-11e6-b1a0-64510650abcf",
  "inputs": [{
    "intent": "action.devices.SYNC"
  }]
}

Click Send the request and verify that your service successfully returns a SYNC response with a valid agentUserId.

Learn more

For more helpful tips on getting the most out of your actions, be sure to check out the rest of the AoG ProTips video series — and share your thoughts with us on Twitter using the hashtag #AoGProTips.

AoG ProTips: Account Linking Verification for Smart Home was originally published in Google Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.