Python Transformation

Executes a Python function provided by the user on a DataFrame connected to its input port. Returns the results of the execution as a DataFrame.

Also returns a Transformer that can be later applied to another DataFrame with a Transform operation.

The Python function that will be executed must:

• be named transform,

• take exactly one argument of type DataFrame,

• return a DataFrame (or data which can be automatically converted to Spark DataFrame: pandas.DataFrame, single value, tuple/list of single values, tuple/list of tuples/lists of single values).

The variables and functions available in the operations’ global scope:

• dataframe() - a function that returns the input DataFrame for this operation. Everytime the input DataFrame changes, the dataframe() returns the updated DataFrame.

• sc - Spark Context

• spark - Spark Session

• sqlContext - SQL Context

Example Python code:

from pyspark.sql.types import Row

def transform(dataframe):
    return spark.createDataFrame(dataframe.rdd.map(lambda row: Row(row.numbers_column*2)))

Since: Seahorse 1.0.0

Input

Output

Parameters

Example

Parameters

def transform(df):
    return df.filter(df.temp > 0.4).sort(df.windspeed, ascending=False)

Input

Output