Georgian College
BDAT 1002
In-class Lab 3 Code ******************************************************************************** ************************ // getting the dataset using wget command wget https://www.dropbox.com/s/biarzqglr11mpx8/german_credit_data.csv // Created HDFS directory hadoop fs -mkdir /BigData // Loading into HDFS hadoop fs -copyFromLocal german_credit_d
...[Show More]
In-class Lab 3 Code ******************************************************************************** ************************ // getting the dataset using wget command wget https://www.dropbox.com/s/biarzqglr11mpx8/german_credit_data.csv // Created HDFS directory hadoop fs -mkdir /BigData // Loading into HDFS hadoop fs -copyFromLocal german_credit_data.csv /BigData/. // start Spark spark-shell --master yarn // import the machine learning packages. import org.apache.spark.ml.classification.RandomForestClassificationModel import org.apache.spark.ml.classification.RandomForestClassifier import org.apache.spark.ml.evaluation.BinaryClassificationEvaluator import org.apache.spark.ml.feature.StringIndexer import org.apache.spark.ml.feature.VectorAssembler import org.apache.spark.ml.tuning.{ ParamGridBuilder, CrossValidator, CrossValidatorModel } import org.apache.spark.ml.{ Pipeline, PipelineStage, PipelineModel } import org.apache.spark.sql.SQLContext import org.apache.spark.sql.types.{StructType, StructField, StringType, IntegerType, Metadata}; // define the Credit Data Schema val sqlContext = new SQLContext(sc) val schema = StructType( StructField("SN", IntegerType, nullable = true) :: StructField("Age", IntegerType, nullable = true) :: StructField("Sex", StringType, nul
[Show Less]
Access Full Document
Instant download after payment
Card Payments
₿
Crypto Accepted
