pyspark.sql.functions.coalesce#

pyspark.sql.functions.coalesce(*cols)[source]#

Returns the first column that is not null.

New in version 1.4.0.

Changed in version 3.4.0: Supports Spark Connect.

Parameters

colsColumn or column name: list of columns to work on.

Returns

Column: value of the first column that is not null.

Examples

>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([(None, None), (1, None), (None, 2)], ("a", "b"))
>>> df.show()
+----+----+
|   a|   b|
+----+----+
|NULL|NULL|
|   1|NULL|
|NULL|   2|
+----+----+

>>> df.select('*', sf.coalesce("a", df["b"])).show()
+----+----+--------------+
|   a|   b|coalesce(a, b)|
+----+----+--------------+
|NULL|NULL|          NULL|
|   1|NULL|             1|
|NULL|   2|             2|
+----+----+--------------+

>>> df.select('*', sf.coalesce(df["a"], lit(0.0))).show()
+----+----+----------------+
|   a|   b|coalesce(a, 0.0)|
+----+----+----------------+
|NULL|NULL|             0.0|
|   1|NULL|             1.0|
|NULL|   2|             0.0|
+----+----+----------------+