The User Flow chart is the best way to easily understand what your users are doing with your LLMs. This chart identifies the most frequent Actions (tasks or questions) done by your users. You can use User Flow to understand how your users sequentially ask questions to your LLM-product and analyze drop-offs or unsuccessful behavior.

Here are examples of the types of questions you can address using Flows:

  • What did users do after asking an information?
  • Which actions do users most frequently take after requesting a text rewrite?
  • What actions lead up to a succesfull answer from the LLM?

Quick start

Creating a User Flow chart involves the same fundamental steps as constructing any other type of chart. Generating a User flow chart is a quick and straightforward process, requiring only a few clicks, with results delivered almost instantaneously.

Step 1: Choose first action

Action forms the foundational element of a User Flow. To analyze user behavior, start by choosing the initial action. This means selecting the specific action after which you want to examine what actions users took subsequently.

Alt Text

Step 2: Choose Steps Before/After Action

Choose a designated number of Steps to identify the actions users took either before or after a particular action.

Alt Text

Step 3: Choose Time window

The Time window is a set duration, for example, 30 minutes, during which you monitor the user’s actions. It highlights what users did within that period after initiating a specific event, like asking for a summary.

Alt Text

Step 4: Choose Filters

The Filter applies to the initial user action you’ve chosen, enabling you to remove or exclude any undesired data from the results. Supported filters are:

TopicThe main theme the user is talking about during the interaction
Data sourceThe external reference or knowledge sources the LLM might use to fetch or validate information
LanguageThe specific language (like English or Spanish) in which the model generates its response
UserThe specific user ID
ActionThe task (or question) the user performs during an interaction
Frustration reasonSpecific reasons a user might be frustrated, such as the response being too wordy (verbosity), the model having insufficient knowledge, the model not understanding the query, the answer lacking clarity, or the answer not delving deep enough
User SatisfactionA metric to assess user satisfaction with the Language Model (LLM) responses, where “useful” is defined as interactions without any detected reasons for frustration. A “Positive” rating is assigned if over 85% of the LLM interactions are useful. A “Neutral” rating applies when 65% to 85% of the conversations are free from detected frustration reasons. A “Negative” rating is given if less than 65% of the interactions are useful, meaning frustration reasons are detected in more than 35% of the interactions

Alt Text

Step 5: Analyze Results

Now you have a detailed overview of your users’ journey, displaying the specific actions they took along with the corresponding percentages. This consolidated view allows you to easily piece together user behavior patterns all in one place.