Build your first Trustworthy AI system

Welcome to the 697 new members this week! nocode.ai now has 18,251 subscribers.

This tutorial walks through a no-code AI experience to monitor models in production. We will use a tool called IBM OpenScale with Auto setup, which makes a model, deploys it, and adds historical data to simulate a production model. OpenScale monitors are set up. Then we will explore the metrics and data.

In this tutorial, you learn how to:

• Provision a Watson OpenScale service

• Use the Insights Dashboard

• Using the Analytics tools

Let’s Dive In! 🤿

Environment preparation

• Software that we will use: IBM OpenScale, tracks and measures outcomes from your AI models, and helps ensure that they remain fair, explainable, and compliant no matter where your models were built or are running.

• How to get access to OpenScale: you can access it here: https://cloud.ibm.com/catalog/services/watson-openscale there is a Free Lite plan to test the service

• Product Documentation: link

Let’s Start! 🏁

This section explains how to provision Watson OpenScale on the IBM Cloud Pak for Data platform. If you're using IBM Cloud, skip to the On IBM Cloud section.

1. In the Cloud Pak for Data instance, go to the (☰) menu, and under Services, click Instances.

2. Find the OpenScale-default instance from the Instances table, and click the three vertical dots to open the action menu. Then, click Open.

   

3. Because this is the first time you are launching OpenScale, you are presented with a welcome message where you can launch the auto setup process. Click Auto setup.

   

4. In the Connect to Watson Machine Learning panel, leave the defaults because you are using the Watson Machine Learning instance deployed in the same cluster. Click Next.

   

5. In the Connect to your database panel, enter the connection details for your local Db2 database (this is the database you provisioned in a previous section of the admin guide), and click Next.

   

The auto setup of a model will take some time to run.

After it completes, you see a message if it succeeded.

6. Click through the Insights dashboard for the deployed models to make sure that the pages load.

7. If you need to give other users access to the OpenScale instance, go to the (☰) menu, and under Services, click Instances.

   

8. Find the OpenScale-default instance from the Instances table, and click the three vertical dots to open the action menu. Then, click Manage access.

Use the Insights Dashboard

To launch the OpenScale service:

9. Go to the (☰) navigation menu, and click Services -> Instances.

   

10. Click the three vertical dots next to the OpenScale instance that your Administrator has provisioned, and click Open.

    

Now, let's interact with the tools.

11. OpenScale loads the Insights Dashboard, which contains tiles for any models being monitored. The GermanCreditRiskModelICP tile is the one you'll use for this tutorial. This tile was configured using the Auto setup script.

12. Click the Insights icon on the left, make sure that you are on the Model monitors tab, and then open the GermanCreditRiskModelICP model tile. Click the 3-dot menu, then click View details.

    

13. Notice the red alert indicators on the various monitors (Fairness, Quality, and Drift). You should see a red indicator under Fairness. Click the Fairness score.

14. Click on the triangle with the ! under Fairness -> Sex. This indicates that there has been an alert for this attribute in the Fairness monitor. Alerts are configurable, based on thresholds for fairness outcomes, which can be set and altered as needed.

By moving your mouse pointer over the trend chart, you can see the values change. Find and click on a time slice where the graph is below the red threshold line. This indicates that the level of fairness is below the predetermined minimum and that some of the transactions at that time are biased.

15. After you click on one of the time periods, you see details of the Fairness monitor, including a bar chart that shows how many females received the "No Risk" outcome versus males. You can click view calculation to see how the fairness score is calculated. Click View payload transactions.

You see a list of transactions. Look for one of the Monitored Group - Female with a Group Bias check mark and a Prediction of "Risk." Click Explain prediction. If the time period on the graph for Fairness Monitoring doesn't contain this element, go back and choose another time period until you can find one. This makes the explanation more interesting.

Note: Each of the individual transactions can be examined to see them in detail. Doing so will cache that transaction as you see later. Be aware that the Explainability feature requires thousands of REST calls to the endpoint using variations of the data that are slightly perturbed. This can require several seconds to complete.

16. On the Explain tab for this individual transaction, you can see the relative weights of the most important features for this prediction. Examine the data, then click the Inspect tab.

17. In the Inspect view of this transaction, you can see the original features that led to this prediction as well as a series of drop-down menus and input boxes that offer the ability to change each feature. You can find which features will change the outcome (in this case, from "Risk" to "No Risk") by clicking Run Analysis. Note that this requires thousands of REST calls to the endpoint with slight perturbations in the data, so it can take a few minutes to run. Click Run Analysis.

In this particular transaction, you see that the presence of a guarantor on the loan is the only thing required to flip the outcome from "Risk" to "No Risk." Other transactions might show a different analysis, so be aware that your results might vary from this. In this example, you can click the drop-down menu for Others on Loan and change to guarantor.

18. Choosing this new value for guarantor exposes a button for Score new values. Click this button.

In this example, you see that the outcome has now been flipped from "Risk" to "No Risk."

19. Go back to the Insights Dashboard page by clicking on the Insights menu icon. Make sure that you are on the Model monitors tab. Now, open the monitor configuration for the GermanCreditRiskModelICP model by clicking the 3-dot menu on the tile, and then click Configure monitors.

20. Click the Endpoints menu, then the Endpoints tab. Use the Endpoints drop-down menu to select Debiased transactions. This is the REST endpoint that offers a debiased version of the credit risk machine learning model based on the features that were configured (that is, Sex and Age). It presents an inference that attempts to remove the bias that has been detected.

You can see code snippets using cURL, the Java language, and Python, which can be used in your scripts or applications.

Similarly, you can choose the Feedback logging endpoint to get code for Feedback Logging. This provides an endpoint for sending fresh test data for ongoing quality evaluation. You can upload feedback data here or work with your developer to integrate the code snippet provided to publish feedback data to your Watson OpenScale database.

Using the Analytics tools

To use the Analytics tools:

21. Click the Insights icon on the left, make sure that you are on the Model monitors tab, and then open the GermanCreditRiskModelICP model tile. Click the 3-dot menu, then click View details.

22. Notice the red alert indicators on the various monitors (Fairness, Quality, and Drift). You should see a red indicator under Fairness. Click the Fairness score.

23. Click Analytics -> Predictions by Confidence. It might take a minute to create the chart. Here, you can see a bar chart that indicates confidence levels and predictions of "Risk" and "No Risk."

24. From this dashboard, click Analytics -> Chart Builder. Here, you can create charts using various Measurements, Features, and Dimensions of your machine learning model. You can see a chart that breaks down Predictions by Confidence.

Note: You might need to click the date range for Past Week or Yesterday to load the data.

25. You can experiment with changing the values and examine the charts that are created.

Congratulations! 🤯 You now know how to connect a model to a monitoring tool and track the main performance metrics to deploy trustworthy AI into production!

Cheers!

Armand 😎