IP Functions are Generally Available, bringing high-performance network analytics to the Lakehouse

IP data is warehousing data

Every firewall, load balancer, VPN, CDN edge, DNS resolver, Kubernetes cluster, and application server all emit a stream of records keyed on one thing: an IP address. For a large enterprise, these streams collectively generate tens of billions of events a day and are the foundation of some of the most valuable analytics an organization runs, including threat detection, fraud investigation, and network observability. Historically, the industry treated these network observability use cases as specialized use cases that required a specialized stack, leading to silos, fragmented governance, and lock-in.

That changes now. Today, IP Functions are available Generally Available. With this launch, IP address analytics becomes a first-class, high-performance SQL workload on the lakehouse, letting security teams parse, enrich, and analyze their highest-volume IP data alongside the rest of their analytics, under one governance model.

Why network analytics used to be painful

In the past, IP addresses were deceptively hard to handle in SQL. An IPv4 address looks like a string but behaves like a 32-bit integer; IPv6 is 128 bits. A CIDR block like 10.0.0.0/8 isn't a value at all - it's a range of 16 million addresses. Asking "is this IP inside that subnet?" is a range containment problem hiding behind a piece of text.

Without native support, teams were forced into one of a few brittle patterns - each of which trades away correctness, performance, or maintainability:

The result was a lose-lose. SQL-only analysts were locked out of basic IP filtering because it required procedural code. Data engineers burned cycles maintaining UDF libraries and CIDR-expansion jobs. Above all, the critical workloads, like enriching billions of events against threat-intelligence and geo-IP tables, would run for over an hour when the business needed answers in minutes. At a petabyte scale, that gap is the difference between catching an intrusion in progress and reading about it in the post-mortem.

Native IP functions, built into SQL

Databricks introduces a complete set of built-in IP functions that make network data a first-class citizen of the lakehouse. They handle IPv4 and IPv6 uniformly, accept both human-readable STRING and compact BINARY representations, understand CIDR notation natively, and are implemented in the engine itself so the optimizer and Photon can accelerate them.

The example below enriches raw network flow logs with threat intelligence and then identifies suspicious source networks that are scanning large numbers of destinations and ports. What previously required custom parsing logic and bespoke IP libraries can now be expressed directly in SQL using native IP and CIDR operations.

No UDF. No regex. No integer gymnastics. It reads like the question the analyst is actually asking.

The functions

The GA release ships the full toolkit needed to parse, normalize, inspect, and join IP data:

Containment and joins

• ip_cidr_contains(cidr, needle) - tests whether an IP address or another CIDR block falls within a CIDR block. This is the single predicate behind CIDR block joins and high-volume filtering, and the function the entire optimization effort centers on.

Parsing and canonicalization

• ip_host(ip) - normalizes an IPv4 or IPv6 address to its standard form (e.g., collapses 2001:0db8:0000::1 to 2001:db8::1).
• ip_cidr(cidr) - produces the canonical representation of a CIDR block.

Inspecting a CIDR

• ip_network(cidr) / ip_network_first(cidr) - returns the first (network) address of a CIDR block.
• ip_network_last(cidr) - returns the last address of a CIDR block.
• ip_prefix_length(cidr) - returns the prefix length (the number after the /).
• ip_version(ip_or_cidr) - returns 4 or 6 so mixed-protocol addresses can be branched on without special-casing

Representation conversion - for performance

• ip_as_binary(ip_or_cidr) - converts an address or CIDR to its canonical, compact binary form (4 bytes for IPv4, 16 for IPv6). Storing and joining on BINARY skips repeated parsing and shrinks storage.
• ip_as_string(ip_or_cidr) - converts a binary representation back to human-readable text for reporting.

Safe variants for messy data

• try_ip_host(ip), try_ip_cidr(cidr), try_ip_as_binary(ip_or_cidr), try_ip_as_string(ip_or_cidr) - identical to their counterparts, but return NULL instead of erroring on invalid input. Essential when ingesting raw logs where a fraction of records are always malformed, so one bad row never fails a billion-row job.

These native functions compose naturally with the rest of SQL, they're available to every SQL user, with no setup, and the optimizer understands them, which is what makes the performance story possible.

Rearc, which helps enterprises develop GenAI, Data, and Cloud platforms, is leveraging the IP Functions to build network observability use cases for large scale customers.

We built a product for a major financial firm that regularly processes more than 30TB of data per day, where performance and cost efficiency are critical. With IP functions native to the Databricks engine, we were able to parse, validate, and join IP data directly in SQL, replacing a previous ad-hoc implementation with something far more elegant and maintainable. Because the functions are built into the engine, we got this without sacrificing the performance our workloads demand at this scale. They've made network analytics on the lakehouse simpler and faster for us to deliver." —Dara Kharabi, Practice Lead, AI & Data at Rearc

Built for petabyte scale: seconds, not minutes

A common IP analytics challenge is finding a single address or sub-CIDR within a much larger range, which is critical for quickly detecting threats, investigating fraud, and monitoring network activity at scale. This use case represents a range join, which engines have historically struggled with because standard join algorithms rely on equality. Databricks’ engine supports an optimized range join for IP addresses via ip_cidr_contains. 

Databricks’s ip_cidr_contains outperforms traditional warehouses on price and speed across all scales of probe (i.e. “needle”) and block (i.e. “haystack”) tables. We benchmarked ip_cidr_contains across five representative scenarios:

The results show that Databricks’ IP Functions’ performance is much stronger than competitors, even at increasing scales. Once the number of probes exceeds 10B IP addresses and the number of blocks goes beyond 1M CIDRs, query speed begins to flatline. 

The cost gap is just as stark. Even as workloads scale, Databricks remains 2x to as much as 6.4x cheaper.

Stanby is quickly saw the value of running their network monitoring use cases directly on the lakehouse, rather than on external systems:

"Databricks' native IP functions let us work with IP and CIDR data directly in SQL. We no longer rely on brittle string parsing or manual bitwise logic. Being able to parse, validate, and join IP data as first-class SQL operations has made this work simpler and more maintainable for our team. It's a natural fit for running network and traffic analytics alongside the rest of our data on the lakehouse."—Stanby, Data Engineering Leader

Ultimately, these highly performant IP Functions enable an entire class of network workloads to be built directly on the lakehouse.

What this unlocks

With fast, native IP functions, entire workloads move onto the lakehouse that previously couldn't live there:

• CIDR enrichment joins at scale - tag every event with GeoIP, ASN, threat-intelligence, or ownership metadata in seconds, so downstream detection and investigation queries run against enriched data.
• High-volume, real-time filtering - "show me every connection from this suspicious /16 in the last 24 hours" becomes an interactive query instead of a batch job.
• Unified IPv4 and IPv6 analytics - mixed-protocol tables work out of the box, so modern traffic is analyzed, not dropped.
• CIDR-in-CIDR matching - check whether an entire subnet falls within another, at the same performance as IP-in-CIDR, for network-topology and policy analysis.
• First class SQL accessibility - analysts get IP filtering and joins with plain SQL, no procedural code or UDF libraries required.

With these functions natively supported on the lakehouse, these network workloads share one governed copy of the data with the rest of the enterprise - no separate specialized system to license, secure, and keep in sync.

Get started today

Native IP functions are available now Generally Available on Databricks Runtime 18.3 and above. 

See the IP functions reference documentation for the full function list and signatures. The network firehose has always been one of your biggest datasets - now it can finally live where the rest of your analytics do.