Basic Examples

This set of basic examples will introduce you to some of the fundamental Seahorse features.

Example 1 - Build a Simple Regression Model

The goal of this exercise is to build a model predicting apartment prices based on 3 features: beds, baths and sq_ft.

The dataset transactions.csv has 5 columns and 1,000 rows (header row and 999 data rows). Each row provides information about the apartment: city, number of bedrooms, number of bathrooms, size of the apartment (in square feet) and its price.

city,beds,baths,sq_ft,price
CityB,4,1,1294,529377
CityC,4,2,1418,574485
CityC,2,1,600,221661
...

To build the model we will split our initial dataset into two parts - training and validation sets. We will train a linear regression model using training set. The model will be scored (against the validation part of the dataset) and report of the scoring will be produced (model performance report).

Create a New Workflow

1. Create an empty workflow:
   • Go to Seahorse main page and click New workflow.
   • Put machinelearning1 in the Name section - setting a meaningful name will help when looking for a workflow in the workflow list.
   • Press the Create button.
   • Start editing the workflow.
2. Set the source of your data:
   • Right-click on the canvas and select a Read DataFrame operation.
   • Click on the created Read DataFrame operation - the panel on the right now shows its parameters.
   • Select a data source from which you want to read the data. In this case, pick transactions.csv. Once the operation gets executed, the dataset will be downloaded from a web address.

   The Simple Regression Model

3. Select features that will be used to train and evaluate model and create a vector column containing selected features’ values:
   • Drag an edge from the output port of the Read DataFrame. Select an Assemble Vector operation.
   • In the INPUT COLUMNS section:
     • Click Edit selection, this will open the selection window for the INPUT COLUMNS parameter.
     • Select Excluding mode in the top-right corner - this will include all values except the ones selected in the features column.
     • Add city and price to the list - we have to exclude the city column, because it does not contain numeric values and price, because it contains the actual results.
   • Put features in the OUTPUT COLUMN parameter - the operation will add a column with this name to the dataset.
4. Split the dataset into training and test sets:
   • Drag an edge from the middle port of the Assemble Vector. Select a Split operation.
   • Put 0.7 in the SPLIT RATIO parameter - we will use 30% of the data for training and 70% for evaluation.
   • Put 0 in the SEED parameter - the split operation itself is not deterministic. You can put any value here - it will split the dataset differently, but in the same proportions.
5. Use the linear regression algorithm:
   • Right-click on the canvas and select Linear Regression operation.
   • In the FEATURES COLUMN section:
     • Click Edit selection, this will open the selection window for the FEATURES COLUMN parameter.
     • Click at Select by name and put features in the text field - this will use the column created in step 3.
   • In the PREDICTION COLUMN section put prediction - this new column will contain predicted values.
   • In the LABEL COLUMN section:
     • Click Edit selection - this will open a selection window for the LABEL COLUMN parameter.
     • Click at Select by name and put price in the text field - price column contains the actual values that we will evaluate our model against.

6. Drag an edge from the output port of the Linear Regression and select a Fit operation. This operation will be used to fit model to the training set.

7. Drag an edge from the output port of the Fit and select a Transform operation. It will transform the dataset using the model fitted in the previous step.

8. Select evaluation method:
   • Right-click on the canvas and select a Regression Evaluator operation.
     • This operation will create an evaluator for regression algorithms, we will use the default evaluation metric (RMSE).
     • Put prediction in the PREDICTION COLUMN parameter.
     • Put price in the LABEL COLUMN parameter.
     • The evaluator will measure the error between price and prediction columns.
   • Drag an edge from the output port of the Regression Evaluator and select an Evaluate operation.
   • Connect the output port of the Transform with the input port of the Evaluate.

import json
from pyspark.sql.types import *

def transform(dataframe):
    def get_fce(js):
        return float(json.loads(js)['Cambridge']['FCE'])

    sqlContext.registerFunction(
      "get_fce", get_fce, FloatType())
    sqlContext.registerDataFrameAsTable(dataframe, "df")
    return sqlContext.sql(
      "SELECT Math, English, get_fce(Certificates) as FCE FROM df")

sc.parallelize([1,2,3,4,5]).map(lambda x: x*x).collect()

sqlContext.registerDataFrameAsTable(dataframe(), "notebook_df")
sqlContext.sql("SELECT FCE FROM notebook_df").toPandas().sample(5)

dataframe().toPandas().sample(5)

import matplotlib.pyplot as plt
%matplotlib inline

df = dataframe().toPandas()
p = df.plot(kind='hist', stacked=True, bins=20)
p.set_ylabel('Frequency')
p.set_xlabel('Score')