The Nebuly SDK enables you to monitor all the requests made to:

Both of them are supported also when using stream or async modes.

The process is straightforward, you just need to:

  • initialize the SDK with your API key
  • include the user_id in your original Google VertexAI method calls.

You can then use the platform to analyze the results and get insights about your LLM users.

​
Chat Models

import nebuly

import vertexai
from vertexai.language_models import ChatModel, InputOutputTextPair

nebuly.init(api_key="<YOUR_NEBULY_API_KEY>")
vertexai.init(project="<YOUR_PROJECT_ID>", location="<YOUR_LOCATION_ID>")

parameters = {
    "temperature": 0.1,  # Temperature controls the degree of randomness in token selection.
    "max_output_tokens": 256,  # Token limit determines the maximum amount of text output.
}

chat_model = ChatModel.from_pretrained("chat-bison@001")
chat = chat_model.start_chat(
    context="My name is Miles. You are an astronomer, knowledgeable about the solar system.",
)

response = chat.send_message(
    "How many planets are there in the solar system?",
    **parameters,
    # Nebuly additional kwargs
    user_id="<YOUR_USER_ID>",
)

You can find a detailed explanation of the allowed nebuly additional keyword arguments below:

​
user_id
string
required

An id or username uniquely identifying the end-user. We recommend hashing their username or email address, in order to avoid sending us any identifying information.

​
nebuly_tags
dict

Tag user interactions by adding key-value pairs using this parameter. Each key represents the tag name, and the corresponding value is the tag value.

For example, if you want to tag an interaction with the model version used to reply to user input, provide it as an argument for nebuly_tags, e.g. {"version": "v1.0.0"}. You have the flexibility to define custom tags, making them available as potential filters on the Nebuly platform.

​
nebuly_api_key
string

You can use this field to temporarily override the Nebuly API key for the selected model call. The interaction will be stored in the project associated with the provided API key.

​
Generate Text Models

import nebuly

import vertexai
from vertexai.language_models import TextGenerationModel

nebuly.init(api_key="<YOUR_NEBULY_API_KEY>")
vertexai.init(project="<YOUR_PROJECT_ID>", location="<YOUR_LOCATION_ID>")

parameters = {
    "temperature": 0.1,  # Temperature controls the degree of randomness in token selection.
    "max_output_tokens": 256,  # Token limit determines the maximum amount of text output.
    "top_p": 0.8,  # Tokens are selected from most probable to least until the sum of their probabilities equals the top_p value.
    "top_k": 40,  # A top_k of 1 means the selected token is the most probable among all tokens.
}

model = TextGenerationModel.from_pretrained("text-bison@001")
response = model.predict(
    "Give me ten interview questions for the role of program manager.",
    **parameters,
    # Nebuly additional kwargs
    user_id="<YOUR_USER_ID>",
)

You can find a detailed explanation of the allowed nebuly additional keyword arguments below:

​
user_id
string
required

An id or username uniquely identifying the end-user. We recommend hashing their username or email address, in order to avoid sending us any identifying information.

​
nebuly_tags
dict

Tag user interactions by adding key-value pairs using this parameter. Each key represents the tag name, and the corresponding value is the tag value.

For example, if you want to tag an interaction with the model version used to reply to user input, provide it as an argument for nebuly_tags, e.g. {"version": "v1.0.0"}. You have the flexibility to define custom tags, making them available as potential filters on the Nebuly platform.

​
nebuly_api_key
string

You can use this field to temporarily override the Nebuly API key for the selected model call. The interaction will be stored in the project associated with the provided API key.

​
Track Traces

To track your google-model traces we currently expose two different methodologies:

  • The chain of models: you can use the context manager integration built in in the nebuly SDK to catch all the model calls. More information can be found in the chain of models section.
  • Raw endpoints: you can directly use the exposed APIs to send the raw interactions and traces to the nebuly platform. You can find an example of usage of the endpoint here, while the formal endpoint definition is available here.