Quickstart guide: Set up Computer Vision in Flow Builder
PreviousHow to set up appointment scheduling in Google Calendar using Flow BuilderNextSet up an identity verification process flow with the Analyze Image step
Last updated
Last updated
In this quickstart guide, we're going to show you how to use the Analyze Image step in to analyze images, detect and decode entities in QR codes and turn recognized bar codes into text.
A
or any other channel from
A workflow to handle the messages -
An (optional)
As long as the image is clear, the Analyze Image step will automatically detect the presence of a bar code or a QR code. Once detected, the Analyze Image step enables you to do the following:
Parse a barcode from an image
Parse a QR code from an image
Setting up the Analyze Image step to detect a barcode? These are the types of barcode that the step can automatically detect:
If one of these barcode types is detected in an image, the step will turn that barcode into text, which will become the output variable.
If none of these barcodes are detected, perhaps because the image doesn't contain a barcode, or because the image is blurry, the output variable will be empty.
The payload of a QR Code is more than just a string of text or numbers. A QR code might carry a structured ‘object’. Currently, the QR code reader enables you to detect the following payload objects:
Assuming that you save the output of the step in a variable called qr_code, you can access the fields using {{qr_code.<field name>}}. For example, when getting the first name and the last name of a contact card, you should use {{qr_code.first_name}} and {{qr_code.last_name}}. In case you want to use the URL in the payload use {{qr_code.url}}, and so on.
Imagine you have a webshop where you sell sneakers. When your customers want to return their sneakers, they need to validate the sneaker's product code. The product code is on the bar code of the package.
Now all you have to do is build a flow that informs the user that they need to send you a photo of this code, waits for the image, sends it through the Analyze Image step, and extracts the data that you need!
Add an Analyze Image step under the flow's trigger. The Analyze Image step has a few parameters. The first parameter is the Image URL. This is where the step will get the image that contains the barcode from
Add {{messageImage}} under Image URL. When a user sends an image on their smartphone, the image will be available in the flow at the URL contained in the {{messageImage}} variable
Next, choose the Bar Code option from the image content to recognize dropdown. Depending on your exact use case, you might want to pick QR Code as the content to recognize which allows for other types of payloads (see the Supported QR codes section above)
Finally, you have to specify the output variable that will carry the payload that’s recognized by the step. Set it to image_content
Click Save and continue to build the remainder of the flow
Add these intents and phrases to a new Intent Knowledge Base in the Knowledge Bases overview. When you are ready, publish the Knowledge Base to start the training process.
After a few minutes, the knowledge base should be ready to use for a Recognize intent step. Go ahead and add it to your flow with the required settings.
Next, you need to decide which action to take based on different outcomes of the intent.
When designing your flow (and your intent knowledge base), try to imagine the conversations that will happen. These are some good starting points:
What would you want to say to a customer if they were greeting you?
What do you want to say if they are asking for an agent?
What do you want to do if they are asking to return the sneakers they just bought?
For each of those scenarios, the flow's response should be different and relevant. To handle each scenario, you can add a Branch step to ‘branch off’ the conversation.
Add the conditions for the branch step. Follow this image for guidance.
When the Recognize Intent step detects a greeting, set up a Send conversations message step that responds with a message such as “Hi! I'm a bot that can help you to return your sneakers. Send us a photo of the bar code here. Let me know if you want me to forward you to an agent instead!”
If the detected intent is either handover or fallback, set up a Send conversations message step that responds with a message such as “Hang on, I’ll connect you to an agent.”
If the detected intent is return, use the Else branch to reply with a message such as “Hi! I'm a bot that can help you to return your sneakers. Send us a photo of the bar code here. Let me know if you want me to forward you to an agent instead!”
For the greeting intent and the return intent, set the flow to await a response - either the photo or a request to be connected to an agent.
At this point, the user might decide to send either a request to be routed to an agent, or they will send a photo containing the bar code.
If the photo is there, it will be stored in the {{messageImage}} variable as discussed before. If the {{messageImage}} variable is empty, it means that the last message did not contain an image. In the case that the step is empty, you can check the intent of the message to check whether the user asked to be forwarded to an agent.
If the step is not empty, proceed with the Analyze Image step.
In this scenario, there is an image in {{messageImage}}.
After putting it through the Analyze Image step, the output variable {{image_content}} should contain the bar code. If this variable is empty, we’ll assume that the photo was unclear or that the image did not contain a bar code at all. In this case, we can inform the end-user and jump back to Wait for response to handle a follow-up photo. Jumping back also means that we can catch requests to be forwarded to an agent.
If the step successfully detected the bar code in the image, we make an HTTP request to our API to create a return form for the selected document. In your own use case, you might want to add in some contact details for the customer so that you can double-check their identity against your customer database.
This concludes the “happy path” of the flow, and so we send out a final confirmation message to inform the customer, such as “Thanks! You'll be sent an email with the return form shortly!”. See the image below for an overview of the steps that come after the Analyze Image step.
If this happens, we need to implement the remaining branch after awaiting the photo. This branch deals with the situation where {{messageImage}} is empty or not set.
Check the intent of the message to see if the user asked to be forwarded. If so, set the flow to Create Inbox ticket so that a human agent can continue supporting the customer. Otherwise, you can instruct the customer with a message such as “It seems that your message did not contain an image. Please take a photo of the barcode on the package and share it here! If you want me to connect you to an agent, let me know!”.
Next, set the flow to jump back to our Wait for response step so that the whole process repeats itself. See the image below for an overview.
Great! That's the Analyze Image step set up!
Here you see an example of a chat that uses the flow we just created. On the first try, the user sent a photo of a barcode that was cropped off on the right side. On the second try, the photo contains the full bar code and so we end up on the "happy path"!
Leverage the machine-learning powered step. Build some flexibility into the flow that feels natural and works robustly under slight variations of incoming messages.
Follow steps 1 and 2 in the using the following data:
