Gemini Enterprise - Documentation

The Gemini Enterprise integration is a self-hosted utility script that reads end-user activity via Cloud Logging, enriches token counts from Cloud Trace, and ingests each prompt/response interaction into Nebuly. The data flow is: Cloud Logging → (Optional BigQuery layer) → Cloud Trace enrichment → Nebuly ingestion The script is incremental and resumable, so you can re-run it safely at any time without creating duplicates. The source repository is available here.

Prerequisites

A Google Cloud project with a Gemini Enterprise app configured.
The {PROJECT_ID} of the Google Cloud project.
The Google Cloud CLI (gcloud) installed and authenticated.
Python ≥ 3.12 and Poetry installed on the machine that runs the script.
A Google Cloud admin who can grant IAM roles and enable audit logging on the Gemini Enterprise app.

Install & run

Clone the integration examples repository, install dependencies, configure your .env file, and run the sync:

git clone https://github.com/nebuly-ai/nebuly-integration-examples.git
cd nebuly-integration-examples/gemini_enterprise
poetry install
cp .env.example .env
# Fill in NEBULY_API_KEY, GCP_PROJECT_ID, GCP_LOCATION, GCP_ENGINE_ID
poetry run python -m gemini_enterprise_sync --from-date 2026-01-01

Follow the guide below to retrieve the necessary credentials and permissions. The first run requires --from-date. Set it to the earliest date you want to import. For the complete environment variable reference, available CLI flags, payload customization and resumable-coverage details, check the project README.

Retrieve Gemini Enterprise Application information

Finding your GCP_PROJECT_ID

Open the Gemini Enterprise console.Click on the Project Dropdown Menu at the top-left of the page (next to the Google Cloud logo) and select the project hosting your gemini enterprise application.Copy the value listed under the ID column (e.g., gemini-enterprise-17828149).

Finding your GCP_ENGINE_ID and GCP_LOCATION

Open the Gemini Enterprise console.Under Apps, copy the value listed under the ID column (e.g., gemini-enterprise-17828149_1782814981868) for your GCP_ENGINE_ID. Copy the value listed under the Location column (e.g., eu) for your GCP_LOCATION.

Setting your GCP Collection (GCP_COLLECTION)

For 99% of Google Cloud setups, this is hardcoded to default_collection. Unless your company has a highly customized multi-tenant setup managed via GCP APIs, you can safely keep this set as:

GCP_COLLECTION=default_collection

Enable audit logs and traces

Before the sync can read interactions, Gemini Enterprise must emit usage/audit logs and agent traces.

Enable required APIs

Enable the APIs the sync depends on:

gcloud services enable \
  logging.googleapis.com \
  cloudtrace.googleapis.com \
  discoveryengine.googleapis.com \
  telemetry.googleapis.com

Enable usage and audit logging

In the Gemini Enterprise console, select your app, click on the Configurations button (gear icon).Head to Observability and toggle on:

Enable logging of prompt inputs and response outputs, to ingest the prompt and response into Nebuly. This is mandatory for the sync to work.
Enable instrumentation of OpenTelemetry traces and logs, to retrieve input/output token counts from Cloud Trace.

This requires the Gemini Enterprise Admin role (roles/discoveryengine.agentspaceAdmin). When enabled, Google Cloud writes the discoveryengine.googleapis.com/gemini_enterprise_user_activity log stream that the sync reads.See Set up usage and audit logs for the full console walkthrough.

Verify logs are flowing

Open Logs Explorer and run:

logName="projects/{PROJECT_ID}/logs/discoveryengine.googleapis.com%2Fgemini_enterprise_user_activity"

Send a test prompt through your Gemini Enterprise app, then confirm new log entries appear within a few minutes.

Set up authentication and permissions

Nebuly secret key → NEBULY_API_KEY

In Nebuly, go to Settings → Projects → View Nebuly keys → Secret keys and copy the secret key for the project you want to sync.

Create a service account and grant read roles

Create a dedicated service account for the sync machine and grant it read-only access to the data sources it needs:

roles/logging.viewer (or roles/logging.privateLogViewer if audit data is private) — logging.logEntries.list
roles/cloudtrace.user — cloudtrace.traces.get
BigQuery source only: roles/bigquery.jobUser + roles/bigquery.dataViewer — bigquery.jobs.create + bigquery.tables.getData

Example (adjust the service account name as needed):

PROJECT_ID="{PROJECT_ID}"

gcloud iam service-accounts create nebuly-gemini-sync \
  --display-name="Nebuly Gemini Enterprise sync"

SA="nebuly-gemini-sync@${PROJECT_ID}.iam.gserviceaccount.com"

gcloud projects add-iam-policy-binding ${PROJECT_ID} \
  --member="serviceAccount:${SA}" --role="roles/logging.viewer"

gcloud projects add-iam-policy-binding ${PROJECT_ID} \
  --member="serviceAccount:${SA}" --role="roles/cloudtrace.user"

If you use BigQuery as the log source, also grant roles/bigquery.jobUser and roles/bigquery.dataViewer on the dataset or project.

gcloud projects add-iam-policy-binding ${PROJECT_ID} \
  --member="serviceAccount:${SA}" --role="roles/bigquery.jobUser"

gcloud projects add-iam-policy-binding ${PROJECT_ID} \
  --member="serviceAccount:${SA}" --role="roles/bigquery.dataViewer"

See Export logs to BigQuery for more details.

Authenticate the machine

Production (recommended): download a JSON key for the service account and set in your .env file:

GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account-key.json

To get the service account key, go to the Service Accounts page, select your project and the newly created service account, and click on the Keys tab. Create a new JSON key and download it.Local / dev: use Application Default Credentials:

gcloud auth application-default login

Store service account keys in a secrets manager. On GKE or Cloud Run, prefer Workload Identity over long-lived JSON keys.

Export logs to BigQuery (recommended)

By default, the sync uses Cloud Logging as the log source. While this is a good option for development and testing, BigQuery is the recommended ongoing log source for production syncs. BigQuery is a more reliable and scalable log storage solution, and it’s much faster to retrieve data from it compared to Cloud Logging. It’s possible to export logs to BigQuery as they arrive using a Cloud Logging sink.

A new sink holds no historical data — it only exports logs written after the sink is created.Cloud Logging retains entries for a limited window (default 30 days); older data cannot be recovered.Plan your backfill before relying on BigQuery alone (see Backfill historical data below).

Create a BigQuery dataset

Create a dataset in the region where you want to store the logs (for example, EU for Europe).

bq mk --dataset --location=EU ${PROJECT_ID}:gemini_enterprise_logs_nebuly

Choose a location that matches your compliance requirements.

Create the Cloud Logging sink

Create a sink with the filter the sync expects:

gcloud logging sinks create gemini-enterprise-logs-nebuly \
  bigquery.googleapis.com/projects/${PROJECT_ID}/datasets/gemini_enterprise_logs_nebuly \
  --log-filter="logName=\"projects/${PROJECT_ID}/logs/discoveryengine.googleapis.com%2Fgemini_enterprise_user_activity\" AND (jsonPayload.serviceTextReply:* OR protoPayload.response.reply:*)"

You can use the --use-partitioned-tables flag to create partitioned tables instead of date-sharded tables. For more details, see Partitioned tables.The filter matches user-activity log entries that contain an assistant reply, whether in jsonPayload or protoPayload form.By default the sink should create a table named discoveryengine_googleapis_com_gemini_enterprise_user_activity_<date> for each day. Check the BigQuery console to verify.

Grant the sink writer identity access

Describe the sink to get its writer identity, then grant it roles/bigquery.dataEditor on the dataset:

SINK_WRITER_IDENTITY=$(gcloud logging sinks describe gemini-enterprise-logs-nebuly --format='value(writerIdentity)')

bq query --use_legacy_sql=false "
GRANT \`roles/bigquery.dataEditor\`
ON SCHEMA \`${PROJECT_ID}\`.gemini_enterprise_logs_nebuly
TO \"${SINK_WRITER_IDENTITY}\";
"

The console often grants this automatically when you are a project Owner; verify it if exports do not appear in BigQuery.

It might take up to 10 minutes for the sink to be created and the data to appear in BigQuery.

Configure the sync for BigQuery

In your .env file, set:

GCP_LOG_SOURCE=bigquery
GCP_BIGQUERY_TABLE=gemini_enterprise_logs_nebuly.discoveryengine_googleapis_com_gemini_enterprise_user_activity_*

Use a _* suffix when the sink creates date-sharded tables (the default for log sinks).
Use a plain table name (no _*) for partitioned tables.
Optionally set GCP_BIGQUERY_LOCATION if your dataset is not in the default region.

Backfill historical data

Because the sink only captures new logs, it’s required to backfill historical data using the logging source before switching to BigQuery.

Leave GCP_LOG_SOURCE=logging (the default) and run a one-time backfill:

poetry run python -m gemini_enterprise_sync --from-date 2026-01-01

Switch to GCP_LOG_SOURCE=bigquery for ongoing runs.

Nebuly deduplication and the script’s resumable coverage mean there is no gap and no duplicates when the sources hand off.

The backfill process can take a while, depending on the amount of data to process.

FAQ

What are the Google Cloud costs associated with this integration? The sync uses Cloud Logging, Cloud Trace and optionally BigQuery.
Costs are based on the amount of data ingested, but should be negligible for most use cases. As of July 2026:

Cloud Logging: Provides 50 GiB of free ingestion per month. Then it’s $0.50 per GiB.
Cloud Trace: Provides 2.5 M spans for free per month. Then it’s $0.20 per 1M spans.
BigQuery: Provides 1 TiB of free query usage per month. Then it’s $6.25 per TiB.

For more details, see the Google Cloud pricing page. Where does the synced data appear?
Interactions imported by the script show up on the Interactions page in your Nebuly project. When will my data appear?
Cloud Logging and trace data can take a few minutes to settle after a user interaction. Once ingested, there can be a delay of up to 30 minutes before data becomes visible in Nebuly. Subsequent updates follow the same polling interval. Is it safe to re-run the script?
Yes. The script maintains resumable coverage in .cache/coverage.json. Re-running it picks up only new data, and Nebuly deduplicates interactions so duplicates are not created. How can I see what would be sent without triggering ingestion?
Pass the --dry-run flag. The script fetches and converts data but skips the POST to Nebuly’s ingestion endpoint, printing a summary instead. Why do some interactions show 0 tokens?
Token counts come from Cloud Trace lookups. If a trace is missing, not yet available, or the lookup fails, the script reports 0 tokens for that interaction. For example, image generation traces do not contain token counts. Why do some interactions show a low output token count?
Google-specific tools, like Google Deep Research, do not expose their token consumption details. The script will report a low output token (just the amount of tokens required to call these tools) count for these interactions.

Resources

Gemini Enterprise Nebuly integration

Full install guide, env-var table, CLI flags, and customization docs

Set up usage and audit logs

Enable Gemini Enterprise usage and audit logging in Google Cloud

Gemini Enterprise traces and observability

Enable agent tracing and view traces in Cloud Trace

Export logs to BigQuery

Configure Cloud Logging sinks to route entries to BigQuery

Nebuly Interaction API

API reference for the Interaction ingestion endpoint

​Prerequisites

​Install & run

​Retrieve Gemini Enterprise Application information

​Enable audit logs and traces

​Set up authentication and permissions

​Export logs to BigQuery (recommended)

​Backfill historical data

​FAQ

​Resources

Gemini Enterprise Nebuly integration

Set up usage and audit logs

Gemini Enterprise traces and observability

Export logs to BigQuery

Nebuly Interaction API

Prerequisites

Install & run

Retrieve Gemini Enterprise Application information

Enable audit logs and traces

Set up authentication and permissions

Export logs to BigQuery (recommended)

Backfill historical data

FAQ

Resources