The Nebuly SDK enables you to monitor all the requests made to:

All of them are supported also when using stream or async mode.

The process is straightforward, you just need to:

  • import the LangChainTrackingHandler from the Nebuly SDK
  • pass the LangChainTrackingHandler to the original langchain method calls

You can then use the platform to analyze the results and get insights about your LLM users.

Chat Models

  • Python

from langchain_openai import ChatOpenAI
from langchain.schema import SystemMessage, HumanMessage
from nebuly.providers.langchain import LangChainTrackingHandler

chat = ChatOpenAI(openai_api_key="<YOUR_OPENAI_API_KEY>")

messages = [
    SystemMessage(
        content="You are a helpful assistant."
    ),
    HumanMessage(content="What are llms?"),
]

callback = LangChainTrackingHandler(
    api_key="<NEBULY_API_KEY>",
    user_id="<USER_ID>",
    # Additional nebuly kwargs...
)

chat.invoke(messages, {"callbacks": [callback]})

You can find a detailed explanation of the allowed nebuly additional keyword arguments below:

user_id
string
required

An id or username uniquely identifying the end-user. We recommend hashing their username or email address, in order to avoid sending us any identifying information.

nebuly_tags
dict

Tag user interactions by adding key-value pairs using this parameter. Each key represents the tag name, and the corresponding value is the tag value.

For example, if you want to tag an interaction with the model version used to reply to user input, provide it as an argument for nebuly_tags, e.g. {"version": "v1.0.0"}. You have the flexibility to define custom tags, making them available as potential filters on the Nebuly platform.

LCEL Chains

  • Python

from langchain.prompts import PromptTemplate
from langchain.schema import StrOutputParser
from langchain_openai import ChatOpenAI
from nebuly.providers.langchain import LangChainTrackingHandler

prompt = PromptTemplate.from_template(
    "What is a good name for a company that makes {product}?"
)
runnable = prompt | ChatOpenAI() | StrOutputParser()

callback = LangChainTrackingHandler(
    api_key="<NEBULY_API_KEY>",
    user_id="<USER_ID>",
    # Additional nebuly kwargs...
)

runnable.invoke({"product": "colorful socks"}, {"callbacks": [callback]})

You can find a detailed explanation of the allowed nebuly additional keyword arguments below:

user_id
string
required

An id or username uniquely identifying the end-user. We recommend hashing their username or email address, in order to avoid sending us any identifying information.

nebuly_tags
dict

Tag user interactions by adding key-value pairs using this parameter. Each key represents the tag name, and the corresponding value is the tag value.

For example, if you want to tag an interaction with the model version used to reply to user input, provide it as an argument for nebuly_tags, e.g. {"version": "v1.0.0"}. You have the flexibility to define custom tags, making them available as potential filters on the Nebuly platform.

Chains

  • Python

from langchain.chains import LLMChain
from langchain.prompts import PromptTemplate
from langchain.prompts.chat import (
    ChatPromptTemplate,
    HumanMessagePromptTemplate,
)
from langchain_openai import ChatOpenAI
from nebuly.providers.langchain import LangChainTrackingHandler

callback = LangChainTrackingHandler(
    api_key="<NEBULY_API_KEY>",
    user_id="<USER_ID>",
    # Additional nebuly kwargs...
)

human_message_prompt = HumanMessagePromptTemplate(
    prompt=PromptTemplate(
        template="What is a good name for a company that makes {product}?",
        input_variables=["product"],
    )
)
chat_prompt_template = ChatPromptTemplate.from_messages([human_message_prompt])
chat = ChatOpenAI(openai_api_key="<YOUR_OPENAI_API_KEY>", temperature=0.9)
chain = LLMChain(llm=chat, prompt=chat_prompt_template)
chain.invoke({"product": "colorful socks"}, {"callbacks": [callback]})

You can find a detailed explanation of the allowed nebuly additional keyword arguments below:

user_id
string
required

An id or username uniquely identifying the end-user. We recommend hashing their username or email address, in order to avoid sending us any identifying information.

nebuly_tags
dict

Tag user interactions by adding key-value pairs using this parameter. Each key represents the tag name, and the corresponding value is the tag value.

For example, if you want to tag an interaction with the model version used to reply to user input, provide it as an argument for nebuly_tags, e.g. {"version": "v1.0.0"}. You have the flexibility to define custom tags, making them available as potential filters on the Nebuly platform.

Retrievers

  • Python

from langchain.chains import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import WebBaseLoader
from langchain_community.vectorstores import FAISS
from langchain_core.prompts.chat import ChatPromptTemplate
from langchain_openai import ChatOpenAI
from langchain_openai import OpenAIEmbeddings

from nebuly.providers.langchain import LangChainTrackingHandler

callback = LangChainTrackingHandler(
    api_key="<NEBULY_API_KEY>",
    user_id="<USER_ID>",
    # Additional nebuly kwargs...
)

loader = WebBaseLoader("https://docs.smith.langchain.com/overview")

docs = loader.load()
embeddings = OpenAIEmbeddings()
text_splitter = RecursiveCharacterTextSplitter()
documents = text_splitter.split_documents(docs)
vector = FAISS.from_documents(documents, embeddings)

prompt = ChatPromptTemplate.from_template(
    """Answer the following question based only on the provided context:

    <context>
    {context}
    </context>

    Question: {input}"""
)

llm = ChatOpenAI(temperature=0)
document_chain = create_stuff_documents_chain(llm, prompt)

retriever = vector.as_retriever()
retrieval_chain = create_retrieval_chain(retriever, document_chain)

retrieval_chain.invoke(
    {"input": "how can langsmith help with testing?"}, {"callbacks": [callback]}
)

You can find a detailed explanation of the allowed nebuly additional keyword arguments below:

user_id
string
required

An id or username uniquely identifying the end-user. We recommend hashing their username or email address, in order to avoid sending us any identifying information.

nebuly_tags
dict

Tag user interactions by adding key-value pairs using this parameter. Each key represents the tag name, and the corresponding value is the tag value.

For example, if you want to tag an interaction with the model version used to reply to user input, provide it as an argument for nebuly_tags, e.g. {"version": "v1.0.0"}. You have the flexibility to define custom tags, making them available as potential filters on the Nebuly platform.

Agents

  • Python

from langchain import hub
from langchain.agents import AgentExecutor, tool, create_openai_functions_agent
from langchain_openai import ChatOpenAI

from nebuly.providers.langchain import LangChainTrackingHandler

callback = LangChainTrackingHandler(
    api_key="<NEBULY_API_KEY>",
    user_id="<USER_ID>",
    # Additional nebuly kwargs...
)

llm = ChatOpenAI(openai_api_key="<YOUR_OPENAI_API_KEY>")

@tool
def get_word_length(word: str) -> int:
    """Returns the number of characters in a word."""
    return len(word)

tools = [get_word_length]

prompt = hub.pull("hwchase17/openai-functions-agent")
agent = create_openai_functions_agent(llm, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools)
agent_executor.invoke(
    {"input": "how can I install langsmith?"},
    {"callbacks": [callback]}
)

You can find a detailed explanation of the allowed nebuly additional keyword arguments below:

user_id
string
required

An id or username uniquely identifying the end-user. We recommend hashing their username or email address, in order to avoid sending us any identifying information.

nebuly_tags
dict

Tag user interactions by adding key-value pairs using this parameter. Each key represents the tag name, and the corresponding value is the tag value.

For example, if you want to tag an interaction with the model version used to reply to user input, provide it as an argument for nebuly_tags, e.g. {"version": "v1.0.0"}. You have the flexibility to define custom tags, making them available as potential filters on the Nebuly platform.