Remove Stop Words

Filters out default English stop words from the input. Null values from the input array are preserved.

This operation is ported from Spark ML.

For a comprehensive introduction, see Spark documentation.

For scala docs details, see org.apache.spark.ml.feature.StopWordsRemover documentation.

Since: Seahorse 1.0.0

Input

Port Type Qualifier Description
0DataFrameThe input DataFrame.

Output

Port Type Qualifier Description
0DataFrameThe output DataFrame.
1TransformerA Transformer that allows to apply the operation on other DataFrames using a Transform.

Parameters

Name Type Description
case sensitive Boolean Whether to do a case sensitive comparison over the stop words.
operate on InputOutputColumnSelector The input and output columns for the operation.

Example

Parameters

Name Value
case sensitive false
operate on one column
input column "raw"
output append new column
output column "removed"

Input

id raw
0 [I,saw,the,red,baloon]
1 [Mary,had,a,little,lamb]

Output

id raw removed
0 [I,saw,the,red,baloon] [saw,red,baloon]
1 [Mary,had,a,little,lamb] [Mary,little,lamb]