How Heroes generate Inspection Reports with VertexAI & AppsScript

[Stefan_QREW]

How I Built an AI-Powered Inspection App for the Avengers

Even superheroes can’t save the universe without efficient business processes! 

In this Video, I walk through how Google’s hidden superpower, AppSheet, helped the Avengers make reporting on Hero Work a breeze.

This video features a clean & easy Inspection App with a Gemini/Vertex integration. Here’s some of the highlights of the build. If any of these seem valuable to your use cases, peek under the hood and retrofit. 

• Data lives in AppSheet Database; Allows two User types to file multiple Inspections on multiple Sites (with mapping support)
• Virtual Columns on the Sites table Averages the severity score from related Inspections and powers a format rule
• Inspections collect an Image and a Severity score. 
• Bot takes that Severity score and converts it to a Text-Based Generative AI prompt.
• A VERY SIMPLE AppsScript function manages the communication between AppSheet and the VertexAI endpoint. 
• AppsScript file has all the Variables you need to update for your own use marked in a dedicated file. EASY.
• VertexAI returns a dramatized “Mission Report” Summary in Markdown format so it can look clean in the AppSheet Markdown supported LongText field. 
• I have a Loading GIF that shows/hides while waiting for the script to finish. 

Obviously, this use-case is fictitious and I have no personal or professional relationship with the Avengers… 

How to copy and run in your environment

In order to get this VertexAI integration running, you’ll need to follow these steps (The UI / location of some of these things may have changed since writing):

GCP Steps

1. Create / Open an Google Cloud Project at https://console.cloud.google.com
1. Take note of the Project ID for this project. You will need it later. 
2. You will need to be broadly permissioned to use the Authentication method I use in the script. 
   
   
2. Attach a Billing Method
1. Search “Billing” in the top bar of the Console. 
2. Add a credit card to the account. 
3. The model I use text-bison is SUPER CHEAP. You will likely not spend more than a dollar experimenting with this. 
   
   
3. Enable the Vertex AI APIs
1. Search “Vertex” in the top bar of the Console.
2. Find the Enable Recommended APIs button. 

Whew… done with that. 

AppsScript Steps

1. Go to my public AppsScript file and Copy it
1. Navigate to the AppsScript File
2. On the top right of the information page, there will be a COPY icon. Press that. 
3. Rename to something local to your environment (You’ll have to search for it in the next step)
   
   
2. Change the variable PROJECT_ID in constants.gs file to the project ID your created in the GCP setup.
    
3. Run the  quickTest() function from the AppsScript Runtime tools (explained in the comments on Code.gs file).
   This will prompt you to authenticate. Once authenticated, the script will store your access token for a while. This is required for running this code from AppSheet.
   
   
4. Update the prompt or the temperature of the AI if you want to explore other outputs 😀

AppSheet Steps

1. Go to my public Sample App and Copy it.
1. Navigate to the link above. Should open the AppSheet Editor for this app
2. Click the Copy Icon in the top left of the Editor UI. 
3. Rename the App to your liking. Copy files & data.
4. You can save to whatever data source makes sense for you. 
2. Retarget the Automation to your version of the AppsScript
1. Open Automation Menu
2. Click on the first Bot listed
3. Click on the Get Summary from VertexAI action step
4. In the settings of that Step, Click on the Script Project file selector
5. Retarget to the script file you just created.  
6. Click the blue Authorize Button
3. Test the app out by adding a new inspection form to any of the Sites

If you enjoyed this demo, had any issues during setup, or actually helped save the world with your adaptation of it, let me know in a comment below!

For more free content, Check out our YouTube Channel. For a big library of Training content prepared by product experts, check out AppSheetTraining.com. 

Do you want a helping hand in unlocking the full power of this technology for your organization? Reach out to us at www.QREWTech.com

[Rifad]

Exciting stuff! AI is revolutionizing the way things are done in the world. I have a variety of AI functions in my apps that are similar to this one, like order breakdown. Now, when a customer sends an order document, all you have to do is upload it to AppSheet. Doesn’t matter what document structure it has. This can extract exactly what I want for my app.

I use nodejs and claude to process and extract each product from the document into line items. This will automatically add line items from the order with product name, quantity, specs, additional notes and so on. Then, a second level AI does a feasibility study and compares the specs with our product database and automatically selects the right product sku if it's feasible. This used to be a manual process, but now it's all done in less than a minute. The future is looking pretty wild!

One awesome feature is that the AI can suggest customized options by combining different products from the database if an item is not found in the database. This is pretty cool because most people wouldn't think of that on their own.

[Stefan_QREW]

@Rifad This is really exciting stuff you're working on. I am especially intrigued with your second level operations comparing against a product DB. 🎉

How are you leveraging LLMs to query/suggest based on data in tabular databases? I've experimented in a few places, but have come up short. Any tips would be much appreciated

@QREW_Cam 

[Rifad]

---

---

@Stefan_QREW wrote:

I am especially intrigued with your second level operations comparing against a product DB. 🎉

---

Everything is brand new for all of us, including myself. So, it's pretty surprising how well things are turning out. There's still so much more to discover, especially when it comes to using AI to automate tasks. 

---

@Stefan_QREW wrote:

How are you leveraging LLMs to query/suggest based on data in tabular databases? I've experimented in a few places, but have come up short. Any tips would be much appreciated

---

I start by using Claude to get a clearer picture of the context window. When sending data to LLM, I make sure to only send the relevant information it needs instead of the entire database. This means combining multiple columns of one row, making it easier for LLM to understand. For example, if I have a product category extracted from level 1 of LLM, I only need to send that category's database in the first level. This makes the process more efficient.