> ## Documentation Index
> Fetch the complete documentation index at: https://docs.wandb.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Trace data loss in worker processes

Weave uploads trace data in background threads to minimize impact on your application's performance. However, you might lose trace data when you use a multiprocessing or task queue system, or a worker process like Celery. This happens if the worker process exits before background threads finish uploading traces.

To prevent data loss in worker processes, ensure that background uploads complete by calling `client.flush()` or `client.finish()` before the worker task completes. This helps in short-lived environments like [AWS Lambda](https://aws.amazon.com/lambda/), [Google Cloud Run](https://cloud.google.com/run), or notebooks that exit right after execution.

These methods have different purposes, so choose the one that fits your environment:

* `weave.flush()`: Provides silent flushing. Recommended when Weave is integrated into worker processes or continuous integration (CI) environments.
* `weave.finish()`: Provides progress feedback with a progress bar or status callbacks. Recommended for interactive scripts or notebooks.

Both methods block until all background uploads complete, which ensures that no trace data is lost when the worker process exits.

The following example shows how to call `client.finish()` from a Celery task to flush traces before the worker returns. Replace `[TEAM-NAME]` with your W\&B team name and `[PROJECT-NAME]` with your W\&B project name:

```python theme={null}
from celery import Celery
import weave

app = Celery('tasks')

@app.task
def process_task(input_data):
    weave.init("[TEAM-NAME]/[PROJECT-NAME]")

    try:
        # Your task logic that creates traces
        result = your_processing_function(input_data)

        # Ensure that all traces are uploaded before the task completes
        weave.finish()

        return result
    finally:
        pass
```

As an alternative, you can use the `with` context manager to call `weave.finish()` automatically on exit:

```python theme={null}
with weave.init("[TEAM-NAME]/[PROJECT-NAME]") as client:
    result = your_processing_function(input_data)

    return result
```

You can also improve the performance of your application with `weave.flush()`. For more information, see [Flushing](/support/weave/articles/long-eval-clean-up-times).

For additional resilience against data loss, consider enabling the [write-ahead log](/weave/guides/tracking/write-ahead-log), which writes trace data to disk before it sends the data to the server.

***

<Badge stroke shape="pill" color="orange" size="md">[Trace Data](/support/weave/tags/trace-data)</Badge><Badge stroke shape="pill" color="orange" size="md">[Performance](/support/weave/tags/performance)</Badge>