Join our product leaders to architect your agentic future We are announcing a new logging feature that is now available in Python User-Defined Function (UDF) in BigQuery. Developers can now use Python’s standard logging library directly inside BigQuery UDFs to emit structured logs seamlessly into Cloud Logging—with zero external plugins or custom connectors required. While UDF logging enables a wide range of operational monitoring and debugging workflows, to demonstrate this feature we are showcasing an example for building fault-tolerant, resilient data pipelines.
In the example below, we are capturing row-level diagnostic telemetry during query execution, so that teams can easily isolate problematic records, trigger real-time alerts, and triage issues via SQL without halting entire workloads. In this post, we’ll show you how to get started with Python UDF logging and walk through an e-commerce event ingestion pipeline as a practical, end-to-end example. Inline validation : The Python UDF parses incoming JSON payloads. Valid records pass to the destination table; corrupted or non-compliant rows trigger logging. error() and return NULL to prevent query failure.
Real-time alerting : A Cloud Monitoring log-based metric notifies your team when error rates breach SLA thresholds. SQL triage & replay : Data engineers query the quarantined payloads in BigQuery using standard SQL over the linked _AllLogs view, diagnose the root cause, and re-ingest recovered data. Although this specific demonstration relies on Python for data validation and transformation, you can seamlessly integrate Python-driven logging and validation into pipelines that perform SQL-based transformations. This UDF parses JSON and checks for a required field.
If parsing fails or the required key is missing, it logs structured diagnostic JSON and returns NULL . In this example, we simulate an e-commerce event ingestion pipeline processing incoming user actions ( click , view , checkout ). Our goal is to extract valid events into our clean analytics table while gracefully quarantining broken or missing-schema payloads. Enable logging in your session with SET @@udf. properties : The query succeeds without interruption while bad rows (2 and 3) are routed to Cloud Logging.
