Category

Knowledge

Voice for Databases

The rise of Voice, the fall of SQL

By Databases, Knowledge, Natural Language, Schema, Voice Search No Comments

Imagine if the only language you needed to talk to your database was English? No SQL. No NoSQL. No tables. Just the information you care about at the tip of your tongue. As natural as asking someone else…

At Aiqudo, we’re building a knowledge retrieval system to do exactly that, all while being personalized to your own domain or industry. (e.g., healthcare, finance, etc…). Take a nurse trying to look up a patient’s medications. Instead of having to manually look through a database, all it takes is a voice command, “Show me medication for Milo”, and they’d receive the appropriate information as shown in Figure 1 that could be displayed as well as spoken back, hands-free, even as the nurse is on-the-go during her busy day. 

 

Structured Healthcare Query

Figure 1: Structured Healthcare Query

Voice is IN, SQL is OUT!

See for yourself …

 

 

Tradeoffs: Full power vs Privacy?

The idea of having a natural language interface for a database may not be new, but it’s far from being a fully solved problem. Some solutions are built on the assumption they can directly interact with the database, and therefore, have full access to your data. The reality is that many companies don’t trust third parties with their most sensitive information. We believe organizations shouldn’t have to make these compromises, which is why designing a privacy-conscious solution was one of our top priorities.

With that in mind, tradeoffs are bound to occur based on whether a partner is willing to share the data, or wants to keep some of the data private. Our approach provides solutions for both options. Not having access to the data can limit the complexity of commands that can be asked with reasonable accuracy. An example of a complex command is  “Show me who the insurers are for Tomas Sauer’s patients,” which requires multiple logical jumps: from Dr. Tomas Sauer, to his patients, then to their insurers. This is a multi-hop query that allows you to navigate indirect connections within your data. On the other hand, not having access to your data can still produce a highly performant product. With the use of a custom schema of your database, we are still able to produce a system that works smoothly with basic commands such as aggregation commands and entity-attribute lookup.

Using Schema to understand structured queries

The core of the system is a semantic parser personalized to your database using a schema-based approach. The job of a semantic parser is to transform natural language commands into a machine-interpretable representation, which in our case is a fully formed database query. Unfortunately, databases aren’t very friendly when you don’t speak their language. By language, I’m talking about the structure or schema of your database. What types of entities exist? What are the properties (or attributes) that relate to these entities? What are synonyms for these entities and attributes? Let’s say we have an entity called “patient”. We want to know if it has data attributes like “medications”, “doctors”, and “birthdate” associated with it. Finally, it is very useful to support synonyms you want linked to the attributes,  like “age” to “birthdate”. This is the type of information that’s required to interact with your database natural voice commands, and without ever having to modify it. We don’t really care about any specific patient named Milo, or how he’s prescribed 325mg of Acetaminophen. All we care about is the schema, i.e., the knowledge that the patients exist in your database and that medications are a property of a patient.

Schema Entities Attributes and Synonyms

Figure 2: Example Schema – Entities, Attributes and Synonyms

Processing a command: Voice to Action

Structured Query Flow

Figure 3: Knowledge Retrieval System – Structured Query Flow

So where does the schema fit into our knowledge retrieval system? Going back to the semantic parser mentioned earlier, we can break the processing phase down to four parts:

  1. Domain Classification: The first part is to classify the domain of a user query. If you’re working with multiple schemas, this will help narrow down the domain for a given query (ex. healthcare, banking, etc.)
  2. Keyword and Policy Extraction: Keyword extraction is where we identify potential references to an entity or attribute in that text you input. Policy Extraction refers to the creation of entity-attribute mappings that define the order in which these terms are processed. For example, with “What’s the phone number of Milo’s doctor?”, we’re working with two different attributes “phone number” and “doctor”. Are we looking for Milo’s phone number or Milo’s doctor’s phone number? Getting this order right is critical. The name “policy” comes from the state-action policies in reinforcement learning which share similarities.
  3. Schema Mapping: At this point we have some idea of what knowledge/information you want returned, but most likely, a database won’t have any idea how to process that information. If you ask for a “doctor” when a database labels it “provider“, then we’re at a dead end. Lucky for us, we have that schema you gave us earlier and this won’t be an issue. We’ll know that “doctor” can be mapped to “provider” based on the similarity in meaning. The synonyms you provide help with this similarity computation, but similarity is not exclusive to the synonyms provided. Think of them as helpful hints to us that help clarify what a given entity or property means, optional but extremely useful. Depending on the domain we’re working with, synonyms may not need to be provided manually, and could be automatically generated.
  4. Structured Query Translation: Finally, we’ve accumulated all the information we need, and we move to the last phase, translating the command into a form your database understands, whether its SQL, Cypher, or some other structured database language. If you prefer to keep your data private, this structured query is what you’d be provided to execute on your own database (using your own credentials, thus maintaining data privacy). In the event you share your data, you’ll get full access to our Actionable Knowledge answers service that includes the target data as well as downstream actions that connect your data to your personalized apps and services. For more information on Actionable Knowledge, check out this blog post

Getting the information you need from your database doesn’t need to be a difficult task if you or your coworker have no idea how to write structured queries. Let us help you make their lives easier by making your data accessible in a language everyone knows.

Kenny Kang, Sunil Patil and Mark Maagdenberg

Voice Announcements

Q Actions for Android: Giving App Notifications a Voice and You More Options

By App Actions, Digital Assistants, Knowledge, Voice Search No Comments

We’re ending a crazy 2020 with something sweet – the release of Q Actions 2.5! In our latest version, we’re proud to announce a couple of unique and useful features: Voice Announcements and new Parameter Options. 

Voice Announcements

Now, app notifications have a Voice! We’ve made it super-simple for your apps to talk to you, and for you to follow up … hands free! The Voice Announcements feature gives you full control – you decide which app notifications are announced, and when. You can select one of our preset time profiles like Work “8:00AM – 5:00PM” or create your own. Specific Voice Announcements for incoming Calls, Texts, WhatsApp, and Twitter allow you to act upon the notification with a follow-on action. We empower users to simply do more with voice. Check out the video below to see hands free Voice Announcements in action.

 

Parameter Options

Q Actions can help users select from among multiple valid options for an action using voice. Our Custom Knowledge knows what actions and content are available on a particular app or service. For example, Q Actions knows an awful lot about popular movies, TV series and music.  So, the next time you feel like watching Star Wars, we can give you a list of titles relevant to the app you’ve selected. Of course, if you know exactly what you want, simply tell Q Actions and it’ll take you straight to that title.

Parameter Options

Parameter Options

Do more with voice hands free! Q Actions is now available on Google Play.

 

Phillip Lam, Kiran Hemaraj and Sunil Patil

Q Actions 1.6.2 just released to App Store!

By App Actions, Artificial Intelligence, Conversation, Digital Assistants, Knowledge, Machine Learning, Natural Language, Voice Search No Comments

New Q Actions version now in the App Store

This version of Q Actions features contextual downstream actions, integration with your calendar, as well as under the bonnet improvements to our matching engines. Q Actions help users power through their day by being more useful and thoughtful.

Contextual Awareness

Q Actions understands the context when performing your actions. Let’s say you call a contact in your phonebook with the command “call Tiffany”. You can then follow-up with the command “navigate to her house”. Q Actions is aware of the context based on your previous command and is able to use that information in a downstream action.


  • say “call Tiffany”
    • then “navigate to her house”

Calendar Integration


Stay on top of your schedule and daily events with the recently added Calendar actions. Need to see what’s coming up next? Just ask “when is my next meeting?” and Q Actions will return a card with all the important event information. Need to quickly schedule something on your calendar? Say “create a new event” and after a few questions, your event is booked. On the go and need to join a video conferencing meeting? Simply say “join my next meeting” and Q Actions will take you directly to your meeting in Google Meet. All you have to do from there is confirm your camera/audio settings and join!

  • “when is my next meeting?”
  • “create a new event”
  • “join my next meeting”

Simply do more with voice! Q Actions is now available on the App Store.

Q Card for Tom Petty

What can you do with that Thing?

By Conversation, Knowledge, User Interface, Voice Search No Comments

Often, when you have something to do, you start by searching for information about a particular Thing. Sometimes, you know exactly what that Thing is, but often, you find the Thing by using information related to it. 

“Who is Taylor Swift?” → Taylor Swift

“Who directed Avatar”  → “James Cameron”

The “Thing” is what we call a Knowledge Entity and something that you can do with that Thing is what we call a Downstream Action. The bond between that Knowledge Entity and the Downstream Action is what we refer to as Actionable Knowledge.

Actionable Knowledge

How do we do this? Our Knowledge database holds information about all kinds of entities such as movies, TV series, athletes, corporations etc. These Entities have rich semantic structure; we have detailed information about the different attributes of these Entities along with the Actions one can perform on those entities. An Action may be generic (watch a show), but can also be explicitly connected to a mobile app or service (watch the show on Disney+). This knowledge allows the user to follow up on an Entity command with an Action. 

For example, asking a question such as “How tall is Tom Brady?”  allows you to get his height i.e., 6’ 4” or 1.93 metres (based on the Locale of who’s asking) since Knowledge captures these important attributes about Tom Brady. Note that these attributes are different for different types of Entities. That is determined by the Schema of the Entity, which allows validation, normalization and transformation of data.

A command like “Who is Tom Brady?” returns a Q Card with information about Tom Brady, as shown below. As there may be multiple entities referring to “Tom Brady”, a popularity measure is computed so that the correct Tom Brady is returned, based on popularity, context and your current session. Popularity is a special attribute that is computed from multiple attributes of the entity. An Entity Card surfaces the various attributes associated with the attribute, such as when Tom Brady was born, how tall and heavy he is, and what sport he plays. There are also attributes that define potential Actions that can follow, so “go to his Instagram” will instantly take you to Tom Brady’s account in the Instagram app. 

Q Card for Tom Brady

Actions are about getting things done! Here’s another example of being able to instantly go from information to Action using Actionable Knowledge.  Asking “Who is Tom Petty?” followed by a command “listen to him on Spotify” will start playing his music. This is a powerful feature that provides a great user experience and rapid Time to Action® .

Q Card for Tom Petty

The three pillars of the Aiqudo’s Q Actions Platform allow us to implement downstream Actions:

  1. Semantically rich Entities in Actionable Knowledge
  2. AI-based Search
  3. Powerful Action execution engine for mobile apps and cloud services

AI Search

We are not limited by just the name of the entity. Our AI-based search allows you to find entities using various attributes of the entity. For example, you can search for stock information by saying “How is Tesla stock doing today?” or “Show me TSLA stock price”.   Aiqudo understands both the corporation name or the stock ticker when it needs to find information on a company’s stock price.  Some apps like Yahoo Finance can only understand the stock ticker; it may not be built to accept the name of the company as an input. Our platform allows us to fill this gap by decoupling action execution from search intent detection. A middle-tier federation module acts as a bridge between intent extraction and Action execution by extracting the right attributes of the Entity returned by the search to those required by the Action execution engine. In the above example it extracts the stockTicker attribute (TSLA),  from the corporation entity retrieved by the search (Tesla) and feeds it to the Action engine. 

Q Card for Tesla Stock

Voila! Job done!

So, what can you do with that Thing? Well, you can instantly perform a meaningful Action on it using the apps on your mobile phone. In the example above, you can jump to Yahoo News to get the latest finance news about Tesla, or go to the stock quote screen within E*Trade, the app you use and trust, to buy Tesla shares and make some money!