AI Data Vision

Azure Machine Learning Prompt Flow is a development tool designed to simplify the development of AI applications that interact with Large Language Models (LLMs). It facilitates the experimentation and deployment of AI applications. Developers can easily test prompt variants and evaluate their performance. The interface also offers endpoint deployment directly from the studio.

In this article, I will explore the features of Prompt Flow through an example of Enterprise Search. We will discuss testing the “Bring Your Own Data Chat QnA” quickstart available in the studio.

Prerequisites

Here are the resources you might require:

1. Azure Machine Learning, which is essential.
2. Azure OpenAI. It’s worth noting that when you deploy an OpenAI model through Azure Machine Learning, it automatically creates an Azure OpenAI resource in your name, even if you already have one.
3. Optional – Azure Cognitive Search: If you plan to use your own data, you will need to create a Vector Index using Cognitive Search (another option is using a Faiss index).

On the Azure Machine Learning homepage, you can see a carousel of Prompt Flow quickstarts, including the one we’re interested in, allowing us to query our own documents (see image below). It is possible to clone the quickstart from this page.

Hands on Prompt Flow

After cloning the project, you will notice a new flow available in the Prompt Flow tab. From this page, you can:

1. Configure “Connections” to Azure OpenAI, vector databases (such as Weaviate or Qdrant), and more…
2. Associate a “Runtime” with your flow(s), either by creating a new one or linking a pre-configured runtime. Microsoft recommends using memory-optimized compute for this purpose.
3. Create a “Vector Index” from local files or from a dataset you’ve previously created. You must specify whether your vector store will be an Azure Cognitive Search Index or a Faiss Index. Note: This feature only supports specific file types : .md, .txt, .html, .py, .doc, .ppt, .pdf, and .xls.

Done with configuring Prompt Flow, you can open the cloned flow, which is presented as follows:

• An interface for configuring the flow, testing prompt variations, and possibly incorporating Python code, such as with LangChain or Semantic Kernel (highlighted in the blue box in the screenshot).
• A pipeline graph that closely resembles Azure Machine Learning Designer (red box).
• A “Chat” button that enables you to simulate a chat experience for the user (green box).

Upon closer examination of the flow, we can see multiple stages that correspond to those observed in the diagram. For the project we’ve cloned, these stages include:

1. “Inputs” / “Outputs“: Establishing a connection with the chat interface.
2. “embed_the_question“: Embedding the user’s question using an embedding model deployed in AOAI (Azure OpenAI).
3. “search_question_from_indexed_docs“: Matching the vector index (either your custom one or the sample index) and retrieving the top_k relevant documents (2 in my case).
4. “generate_prompt_context“: Utilizing the retrieved content to generate the prompt context.
5. “prompt_variants“: Testing prompt variations.
6. “chat_with_context“: Sending the generated response back to the chat interface.

Testing Prompts and Plugins

Within the “Prompt_variants” box you can do some prompt engineering and test different prompts. For instance, if you generated 3 prompts and want to see how it influenced the response, you can “Run” (button on the upper right corner) your flow and it will executes the entire flow you built.

If you take a look at the diagram after the process, it now indicates it completed 3 times each step. You’ll be able to “View full output” (corresponding button) step by step for each prompt variant.

Once your flow meets your requirements and passes all your tests, you can deploy an endpoint directly from the Prompt Flow interface. This endpoint will enable you to seamlessly integrate your development into applications that utilize generative AI.

Prompt Flow also enables testing of plugins through the use of the Semantic Kernel. In the following documentation, Microsoft explains the integration and testing of plugins using the example of the MathPlugin to address mathematical problems : Evaluate your plugins with Prompt flow.

If we want to delve further into testing, there is an “Evaluation Flow” feature available to assess the outputs against specific criteria. The evaluation is based on the execution results but can also make use of an optional dataset containing ground truth data.

A performance measure of the flow based on individual scores is then calculated and will be used to evaluate your flow. In the image below (from the documentation), there is an accuracy score derived from combining individual scores. Microsoft provides documentation on this topic: Evaluation flow in Prompt flow (preview).

Prompt Flow VS Code Extension

Prompt Flow is also available as an extension to Visual Studio Code. Microsoft documentation on Prompt Flow and its integration into VSC is available here : Prompt flow documentation.