Nebuly’s SDK automatically detects the behaviours of your LLM users. In particular, the concepts we use to build your users data model are as follows:

  • Interactions: An interaction is a data point that comprehends the user input prompt and the answer of the LLM.
  • Actions: An action is the task (or question) the user performs during an interaction. Examples of actions are “ask to summarize”, “ask to change tone”, “ask to write again” or “copy and paste” the answer of the LLM, etc
  • Users: Users are the ones performing the actions.
  • Properties: Properties are attributes that define the specifics of an interaction, an action or a user. Examples of properties are: topic of an interaction (legal, finance, marketing, etc), style, tone, language, satisfaction, frustration, etc

List of actions

Here is the list of user actions that Nebuly automatically detects:

Information RetrievalUsers pose direct questions, with the model sourcing answers from its memory or external RAG sources
Content CreationUsers prompt the model to generate fictional text, unrelated to information retrieval
Summary RequestUsers seek concise summaries of extensive articles or documents
Clarification RequestUsers ask for further explanations, elaborations, or details on a prior query
Alternative AnswersUsers request the model to provide varied responses to a previous query for selection
Translation RequestUsers request language translations
Improvement RequestUsers instruct the model to enhance the quality of specific text or make designated modifications (e.g., word replacements, character renaming)
List CreationUsers direct the model to present text in a bullet-point format
Styling RequestUsers specify a desired style for the model to format the target text
Coding AssistanceUsers seek guidance on coding or debugging issues
Problem SolvingUsers ask for solutions to mathematical equations or general problems
Prompt ElaborationUsers provide supplementary information to clarify a previous query
Answer AcceptanceUsers convey satisfaction with a prior response
Error ReportingUsers report mistakes to the LLM
Question ReiterationUsers repeat a previously posed question
Drop OffUsers don’t pose a follow-up to the LLM within a set time frame (default is 30 minutes)

List of properties

Here is the list of properties of each user interaction that Nebuly automatically detects:

TopicThe main theme the user is talking about during the interaction
Text FormatThe structural layout of the model’s response, such as paragraph, bullet points, article, email, tweet, blogpost, notes, etc
Text StyleThe tone of the response, indicating if it’s formal, casual, humorous, etc.
VerbosityThe length of the response, indicating whether it’s concise or detailed
Text LanguageThe specific language (like English or Spanish) in which the model generates its response
User SatisfactionA measure indicating whether the user found the LLM’s response satisfactory or not
Frustration CausesSpecific reasons a user might be frustrated, such as the response being too wordy (verbosity), the model having insufficient knowledge, the model not understanding the query, the answer lacking clarity, or the answer not delving deep enough
LatencyThe time taken between the user’s query and the LLM’s response
Trace CostThe computational cost associated with generating a response
RAG SourcesThe external reference or knowledge sources the LLM might use to fetch or validate information