User properties
Nebuly’s SDK automatically detects the behaviours of your LLM users. In particular, the concepts we use to build your users data model are as follows:
- Interactions: An interaction is a data point that comprehends the user input prompt and the answer of the LLM.
- Actions: An action is the task (or question) the user performs during an interaction. Examples of actions are “ask to summarize”, “ask to change tone”, “ask to write again” or “copy and paste” the answer of the LLM, etc
- Users: Users are the ones performing the actions.
- Properties: Properties are attributes that define the specifics of an interaction, an action or a user. Examples of properties are: topic of an interaction (legal, finance, marketing, etc), style, tone, language, satisfaction, frustration, etc
List of actions
Here is the list of user actions that Nebuly automatically detects:
| Action | Description |
|---|---|
| Information Retrieval | Users pose direct questions, with the model sourcing answers from its memory or external RAG sources |
| Content Creation | Users prompt the model to generate fictional text, unrelated to information retrieval |
| Summary Request | Users seek concise summaries of extensive articles or documents |
| Clarification Request | Users ask for further explanations, elaborations, or details on a prior query |
| Alternative Answers | Users request the model to provide varied responses to a previous query for selection |
| Translation Request | Users request language translations |
| Improvement Request | Users instruct the model to enhance the quality of specific text or make designated modifications (e.g., word replacements, character renaming) |
| List Creation | Users direct the model to present text in a bullet-point format |
| Styling Request | Users specify a desired style for the model to format the target text |
| Coding Assistance | Users seek guidance on coding or debugging issues |
| Problem Solving | Users ask for solutions to mathematical equations or general problems |
| Prompt Elaboration | Users provide supplementary information to clarify a previous query |
| Answer Acceptance | Users convey satisfaction with a prior response |
| Error Reporting | Users report mistakes to the LLM |
| Question Reiteration | Users repeat a previously posed question |
| Drop Off | Users don’t pose a follow-up to the LLM within a set time frame (default is 30 minutes) |
List of properties
Here is the list of properties of each user interaction that Nebuly automatically detects:
| Property | Description |
|---|---|
| Topic | The main theme the user is talking about during the interaction |
| Text Format | The structural layout of the model’s response, such as paragraph, bullet points, article, email, tweet, blogpost, notes, etc |
| Text Style | The tone of the response, indicating if it’s formal, casual, humorous, etc. |
| Verbosity | The length of the response, indicating whether it’s concise or detailed |
| Text Language | The specific language (like English or Spanish) in which the model generates its response |
| User Satisfaction | A measure indicating whether the user found the LLM’s response satisfactory or not |
| Frustration Causes | Specific reasons a user might be frustrated, such as the response being too wordy (verbosity), the model having insufficient knowledge, the model not understanding the query, the answer lacking clarity, or the answer not delving deep enough |
| Latency | The time taken between the user’s query and the LLM’s response |
| Trace Cost | The computational cost associated with generating a response |
| RAG Sources | The external reference or knowledge sources the LLM might use to fetch or validate information |