Monitoring and usage

• 1: Track user activity with audit logs
• 2: Use Prometheus monitoring
• 3: Configure Slack alerts
• 4: View organization dashboard

1 - Track user activity with audit logs

Use W&B audit logs to track user activity within your organization and to conform to your enterprise governance requirements. Audit logs are available in JSON format. Refer to Audit log schema.

How to access audit logs depends on your W&B platform deployment type:

After fetching audit logs, you can analyze them using tools like Pandas, Amazon Redshift, Google BigQuery, or Microsoft Fabric. Some audit log analysis tools do not support JSON; refer to the documentation for your analysis tool for guidelines and requirements for transforming the JSON-formatted audit logs before analysis.

Audit log schema

This table shows all keys which may appear in an audit log entry, ordered alphabetically. Depending on the action and the circumstances, a specific log entry may include only a subset of the possible fields.

Personally identifiable information (PII)

Personally identifiable information (PII), such as email addresses and the names of projects, teams, and reports, is available only using the API endpoint option.

• For Self-managed and Dedicated Cloud, an organization admin can exclude PII when fetching audit logs.
• For SaaS Cloud, the API endpoint always returns relevant fields for audit logs, including PII. This is not configurable.

Fetch audit logs

An organization or instance admin can fetch the audit logs for a W&B instance using the Audit Logging API, at the endpoint audit_logs/.

1. Determine the correct API endpoint for your instance:

   • Self-managed: <wandb-platform-url>/admin/audit_logs
   • Dedicated Cloud: <wandb-platform-url>/admin/audit_logs
   • SaaS Cloud (Enterprise required): https://api.wandb.ai/audit_logs

   In proceeding steps, replace <API-endpoint> with your API endpoint.

2. Construct the full API endpoint from the base endpoint, and optionally include URL parameters:

   • anonymize: if set to true, remove any PII; defaults to false. Refer to Exclude PII when fetching audit logs. Not supported for SaaS Cloud.

   • numDays: logs will be fetched starting from today - numdays to most recent; defaults to 0, which returns logs only for today. For SaaS Cloud, you can fetch audit logs from a maximum of 7 days in the past.

   • startDate: an optional date with format YYYY-MM-DD. Supported only on SaaS Cloud.

     startDate and numDays interact:

     • If you set both startDate and numDays, logs are returned from startDate to startDate + numDays.
     • If you omit startDate but include numDays, logs are returned from today to numDays.
     • If you set neither startDate nor numDays, logs are returned for today only.

3. Execute an HTTP GET request on the constructed fully qualified API endpoint using a web browser or a tool like Postman, HTTPie, or cURL.

The API response contains new-line separated JSON objects. Objects will include the fields described in the schema, just like when audit logs are synced to an instance-level bucket. In those cases, the audit logs are located in the /wandb-audit-logs directory in your bucket.

Use basic authentication

To use basic authentication with your API key to access the audit logs API, set the HTTP request’s Authorization header to the string Basic followed by a space, then the base-64 encoded string in the format username:API-KEY. In other words, replace the username and API key with your values separated with a : character, then base-64-encode the result. For example, to authorize as demo:p@55w0rd, the header should be Authorization: Basic ZGVtbzpwQDU1dzByZA==.

Exclude PII when fetching audit logs

For Self-managed and Dedicated Cloud, a W&B organization or instance admin can exclude PII when fetching audit logs. For SaaS Cloud, the API endpoint always returns relevant fields for audit logs, including PII. This is not configurable.

To exclude PII, pass the anonymize=true URL parameter. For example, if your W&B instance URL is https://mycompany.wandb.io and you would like to get audit logs for user activity within the last week and exclude PII, use an API endpoint like:

https://mycompany.wandb.io/admin/audit_logs?numDays=7&anonymize=true.

Actions

This table describes possible actions that can be recorded by W&B, sorted alphabetically.

1: On SaaS Cloud, audit logs are not collected for:

• Open or Public projects.
• The report:read action.
• User actions which are not tied to a specific organization.

2 - Use Prometheus monitoring

Use Prometheus with W&B Server. Prometheus installs are exposed as a kubernetes ClusterIP service.

Follow the procedure below to access your Prometheus metrics endpoint (/metrics):

1. Connect to the cluster with Kubernetes CLI toolkit, kubectl. See kubernetes’ Accessing Clusters documentation for more information.

2. Find the internal address of the cluster with:

   kubectl describe svc prometheus
   

3. Start a shell session inside your container running in your Kubernetes cluster with kubectl exec. Hit the endpoint at <internal address>/metrics.

   Copy the command below and execute it in your terminal and replace <internal address> with your internal address:

   kubectl exec <internal address>/metrics
   

A test pod starts, which you can exec into just to access anything in the network:

kubectl run -it testpod --image=alpine bin/ash --restart=Never --rm

From there you can choose to keep access internal to the network or expose it yourself with a kubernetes nodeport service.

3 - Configure Slack alerts

Integrate W&B Server with Slack.

Create the Slack application

Follow the procedure below to create a Slack application.

1. Visit https://api.slack.com/apps and select Create an App.

   

2. Provide a name for your app in the App Name field.

3. Select a Slack workspace where you want to develop your app in. Ensure that the Slack workspace you use is the same workspace you intend to use for alerts.

   

Configure the Slack application

1. On the left sidebar, select OAth & Permissions.

   

2. Within the Scopes section, provide the bot with the incoming_webhook scope. Scopes give your app permission to perform actions in your development workspace.

   For more information about OAuth scopes for Bots, see the Understanding OAuth scopes for Bots tutorial in the Slack API documentation.

   

3. Configure the Redirect URL to point to your W&B installation. Use the same URL that your host URL is set to in your local system settings. You can specify multiple URLs if you have different DNS mappings to your instance.

   

4. Select Save URLs.

5. You can optionally specify an IP range under Restrict API Token Usage, allow-list the IP or IP range of your W&B instances. Limiting the allowed IP address helps further secure your Slack application.

Register your Slack application with W&B

1. Navigate to the System Settings or System Console page of your W&B instance, depending on your deployment

2. Depending on the System page you are on follow one of the below options:

   • If you are in the System Console: go to Settings then to Notifications

     

   • If you are in the System Settings: toggle the Enable a custom Slack application to dispatch alerts to enable a custom Slack application

     

3. Supply your Slack client ID and Slack secret then click Save. Navigate to Basic Information in Settings to find your application’s client ID and secret.

4. Verify that everything is working by setting up a Slack integration in the W&B app.

4 - View organization dashboard

View organization usage of W&B

Use the organization dashboard to get a holistic view of your organization’s usage of W&B. The dashboard is organized by tab.

• Users: lists details about each user, including their name, email, teams, roles, and last activity.
• Service accounts: Lists details about service accounts and allows you to create service accounts.
• Activity: lists details about each user’s activity.
• Teams: lists details about each team, including the number of users and tracked hours, and allows an admin to join a team.
• Billing: ssummarizes your organization’s charges, allows you to run and export billing reports, and shows details about your license, such as when it expires.
• Settings: allows you to configure custom roles and settings related to privacy and authentication.

View the status of a user

The Users tab lists all users, along with data about each user. The Last Active column shows whether a user has accepted an invitation and the user’s current status:

• Invite pending: Admin has sent invite but user has not accepted invitation.
• Active: User has accepted the invite and created an account.
• -: The user was previously active but has not been active in the last 6 months.
• Deactivated: Admin has revoked access of the user.

To sort the list of users by activity, click the Last Active column heading.

View and share how your organization uses W&B

From the Users tab, view details about how your organization uses W&B in CSV format.

1. Click the action ... menu next to the Invite new user user button.
2. Click Export as CSV. The CSV files that downloads lists details about each user of an organization, such as their user name and email address, the time they were last active, their roles, and more.

View user activity

In the Users tab, use the Last Active column to get an Activity summary of an individual user.

1. To sort the list of users by Last Active, click the column name.
2. To see details about a user’s last activity, hover your mouse over the Last Active field for the user. A tooltip appears that shows when the user was added and how many total days the user has been active.

A user is active if they:

• log in to W&B.
• view any page in the W&B App.
• log runs.
• use the SDK to track an experiment.
• interact with the W&B Server in any way.

View active users over time

Use the plots in the Activity tab to get an aggregate view of how many users have been active over time

1. Click the Activity tab.
2. The Total active users plot shows how many users have been active in a period of time (defaults to 3 months).
3. The Users active over time plot shows the fluctuation of active users over a period of time (defaults to 6 months). Hover your mouse over a pointo to see the number of users on that date.

To change the period of time for a plot, use the drop-down. You can select:

• Last 30 days
• Last 3 months
• Last 6 months
• Last 12 months
• All time