WeightHotEncoderEstimator

A weighted one-hot encoder that maps a column of category indices to a column of vectors, with at most a single weight value per row that indicates the input category index. For example with 5 categories and a weight of 0.3, an input value of 2.0 would map to an output vector of [0.0, 0.0, 0.3, 0.0]. The last category is not included by default (configurable via dropLast), because it makes the vector entries linearly dependent. So an input value of 4.0 maps to [0.0, 0.0, 0.0, 0.0].

When handleInvalid is configured to 'keep', an extra "category" indicating invalid values is added as last category. So when dropLast is true, invalid values are encoded as all-zeros vector.

A group encoding is also possible. In this case there can be multiple weight values, with one weight value per row of the group. All rows of a group have the same vector. The weight values of a group can further be weighted by the group size.

Annotations: @Since( "2.3.0" )
Note: When encoding multi-column by using inputCols and outputCols params, input/output cols come in pairs, specified by the order in the arrays, and each pair is treated independently.
See also: StringIndexer for converting categorical values into category indices

Linear Supertypes

DefaultParamsWritable, MLWritable, WeightHotEncoderBase, HasOutputCols, HasInputCols, HasHandleInvalid, Estimator[WeightHotEncoderModel], PipelineStage, Logging, Params, Serializable, Serializable, Identifiable, AnyRef, Any

Instance Constructors

new WeightHotEncoderEstimator()

Annotations
@Since( "2.3.0" )
new WeightHotEncoderEstimator(uid: String)

Annotations
@Since( "2.3.0" )

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def $[T](param: Param[T]): T

Attributes
protected
Definition Classes
Params
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def asInstanceOf[T0]: T0

Definition Classes
Any
final def clear(param: Param[_]): WeightHotEncoderEstimator.this.type

Definition Classes
Params
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
def copy(extra: ParamMap): WeightHotEncoderEstimator

Definition Classes
WeightHotEncoderEstimator → Estimator → PipelineStage → Params
Annotations
@Since( "2.3.0" )
def copyValues[T <: Params](to: T, extra: ParamMap): T

Attributes
protected
Definition Classes
Params
final def defaultCopy[T <: Params](extra: ParamMap): T

Attributes
protected
Definition Classes
Params
final val dropLast: BooleanParam

Whether to drop the last category in the encoded vector (default: true)
Whether to drop the last category in the encoded vector (default: true)

Definition Classes
WeightHotEncoderBase
Annotations
@Since( "2.3.0" )
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def explainParam(param: Param[_]): String

Definition Classes
Params
def explainParams(): String

Definition Classes
Params
final def extractParamMap(): ParamMap

Definition Classes
Params
final def extractParamMap(extra: ParamMap): ParamMap

Definition Classes
Params
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
def fit(dataset: Dataset[_]): WeightHotEncoderModel

Definition Classes
WeightHotEncoderEstimator → Estimator
Annotations
@Since( "2.3.0" )
def fit(dataset: Dataset[_], paramMaps: Array[ParamMap]): Seq[WeightHotEncoderModel]

Definition Classes
Estimator
Annotations
@Since( "2.0.0" )
def fit(dataset: Dataset[_], paramMap: ParamMap): WeightHotEncoderModel

Definition Classes
Estimator
Annotations
@Since( "2.0.0" )
def fit(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): WeightHotEncoderModel

Definition Classes
Estimator
Annotations
@Since( "2.0.0" ) @varargs()
final def get[T](param: Param[T]): Option[T]

Definition Classes
Params
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
final def getDefault[T](param: Param[T]): Option[T]

Definition Classes
Params
def getDropLast: Boolean

Definition Classes
WeightHotEncoderBase
Annotations
@Since( "2.3.0" )
def getGroupCols: Array[String]

Definition Classes
WeightHotEncoderBase
def getGroupWeighting: String

Definition Classes
WeightHotEncoderBase
final def getHandleInvalid: String

Definition Classes
HasHandleInvalid
final def getInputCols: Array[String]

Definition Classes
HasInputCols
final def getOrDefault[T](param: Param[T]): T

Definition Classes
Params
final def getOutputCols: Array[String]

Definition Classes
HasOutputCols
def getParam(paramName: String): Param[Any]

Definition Classes
Params
def getWeights: Array[Double]

Definition Classes
WeightHotEncoderBase
final val groupCols: StringArrayParam

The columns to group by if a group encoding should be used.
The columns to group by if a group encoding should be used.

Definition Classes
WeightHotEncoderBase
final val groupWeighting: Param[String]

The group weighting to use.
The group weighting to use.
"equi" means dividint by the group size "sqrt" means dividing by the square root of the group size "one" means no division
Default: "sqrt"

Definition Classes
WeightHotEncoderBase
val handleInvalid: Param[String]

Param for how to handle invalid data during transform().
Param for how to handle invalid data during transform(). Options are 'keep' (invalid data presented as an extra categorical feature) or 'error' (throw an error). Note that this Param is only used during transform; during fitting, invalid data will result in an error. Default: "error"

Definition Classes
WeightHotEncoderBase → HasHandleInvalid
Annotations
@Since( "2.3.0" )
final def hasDefault[T](param: Param[T]): Boolean

Definition Classes
Params
def hasParam(paramName: String): Boolean

Definition Classes
Params
def hashCode(): Int

Definition Classes
AnyRef → Any
def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

Attributes
protected
Definition Classes
Logging
def initializeLogIfNecessary(isInterpreter: Boolean): Unit

Attributes
protected
Definition Classes
Logging
final val inputCols: StringArrayParam

Definition Classes
HasInputCols
final def isDefined(param: Param[_]): Boolean

Definition Classes
Params
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
final def isSet(param: Param[_]): Boolean

Definition Classes
Params
def isTraceEnabled(): Boolean

Attributes
protected
Definition Classes
Logging
def log: Logger

Attributes
protected
Definition Classes
Logging
def logDebug(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logDebug(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logName: String

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
final val outputCols: StringArrayParam

Definition Classes
HasOutputCols
lazy val params: Array[Param[_]]

Definition Classes
Params
def save(path: String): Unit

Definition Classes
MLWritable
Annotations
@Since( "1.6.0" ) @throws( ... )
final def set(paramPair: ParamPair[_]): WeightHotEncoderEstimator.this.type

Attributes
protected
Definition Classes
Params
final def set(param: String, value: Any): WeightHotEncoderEstimator.this.type

Attributes
protected
Definition Classes
Params
final def set[T](param: Param[T], value: T): WeightHotEncoderEstimator.this.type

Definition Classes
Params
final def setDefault(paramPairs: ParamPair[_]*): WeightHotEncoderEstimator.this.type

Attributes
protected
Definition Classes
Params
final def setDefault[T](param: Param[T], value: T): WeightHotEncoderEstimator.this.type

Attributes
protected
Definition Classes
Params
def setDropLast(value: Boolean): WeightHotEncoderEstimator.this.type

Annotations
@Since( "2.3.0" )
def setGroupCols(value: Array[String]): WeightHotEncoderEstimator.this.type
def setGroupWeighting(value: String): WeightHotEncoderEstimator.this.type
def setHandleInvalid(value: String): WeightHotEncoderEstimator.this.type

Annotations
@Since( "2.3.0" )
def setInputCols(values: Array[String]): WeightHotEncoderEstimator.this.type

Annotations
@Since( "2.3.0" )
def setOutputCols(values: Array[String]): WeightHotEncoderEstimator.this.type

Annotations
@Since( "2.3.0" )
def setWeights(value: Array[Double]): WeightHotEncoderEstimator.this.type
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toString(): String

Definition Classes
Identifiable → AnyRef → Any
def transformSchema(schema: StructType): StructType

Definition Classes
WeightHotEncoderEstimator → PipelineStage
Annotations
@Since( "2.3.0" )
def transformSchema(schema: StructType, logging: Boolean): StructType

Attributes
protected
Definition Classes
PipelineStage
Annotations
@DeveloperApi()
val uid: String

Definition Classes
WeightHotEncoderEstimator → Identifiable
Annotations
@Since( "2.3.0" )
def validateAndTransformSchema(schema: StructType, dropLast: Boolean, keepInvalid: Boolean): StructType

Attributes
protected
Definition Classes
WeightHotEncoderBase
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final val weights: DoubleArrayParam

The weight to use instead of 1.0 for hot encoding, for each column
The weight to use instead of 1.0 for hot encoding, for each column

Definition Classes
WeightHotEncoderBase
def write: MLWriter

Definition Classes
DefaultParamsWritable → MLWritable

Related Docs: object WeightHotEncoderEstimator | package feature

class WeightHotEncoderEstimator extends Estimator[WeightHotEncoderModel] with WeightHotEncoderBase with DefaultParamsWritable

Instance Constructors

new WeightHotEncoderEstimator()

new WeightHotEncoderEstimator(uid: String)

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def $[T](param: Param[T]): T

final def ==(arg0: Any): Boolean

final def asInstanceOf[T0]: T0

final def clear(param: Param[_]): WeightHotEncoderEstimator.this.type

def clone(): AnyRef

def copy(extra: ParamMap): WeightHotEncoderEstimator

def copyValues[T <: Params](to: T, extra: ParamMap): T

final def defaultCopy[T <: Params](extra: ParamMap): T

final val dropLast: BooleanParam

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def explainParam(param: Param[_]): String

def explainParams(): String

final def extractParamMap(): ParamMap

final def extractParamMap(extra: ParamMap): ParamMap

def finalize(): Unit

def fit(dataset: Dataset[_]): WeightHotEncoderModel

def fit(dataset: Dataset[_], paramMaps: Array[ParamMap]): Seq[WeightHotEncoderModel]

def fit(dataset: Dataset[_], paramMap: ParamMap): WeightHotEncoderModel

def fit(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): WeightHotEncoderModel

final def get[T](param: Param[T]): Option[T]

final def getClass(): Class[_]

final def getDefault[T](param: Param[T]): Option[T]

def getDropLast: Boolean

def getGroupCols: Array[String]

def getGroupWeighting: String

final def getHandleInvalid: String

final def getInputCols: Array[String]

final def getOrDefault[T](param: Param[T]): T

final def getOutputCols: Array[String]

def getParam(paramName: String): Param[Any]

def getWeights: Array[Double]

final val groupCols: StringArrayParam

final val groupWeighting: Param[String]

val handleInvalid: Param[String]

final def hasDefault[T](param: Param[T]): Boolean

def hasParam(paramName: String): Boolean

def hashCode(): Int

def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

def initializeLogIfNecessary(isInterpreter: Boolean): Unit

final val inputCols: StringArrayParam

final def isDefined(param: Param[_]): Boolean

final def isInstanceOf[T0]: Boolean

final def isSet(param: Param[_]): Boolean

def isTraceEnabled(): Boolean

def log: Logger

def logDebug(msg: ⇒ String, throwable: Throwable): Unit

def logDebug(msg: ⇒ String): Unit

def logError(msg: ⇒ String, throwable: Throwable): Unit

def logError(msg: ⇒ String): Unit

def logInfo(msg: ⇒ String, throwable: Throwable): Unit

def logInfo(msg: ⇒ String): Unit

def logName: String

def logTrace(msg: ⇒ String, throwable: Throwable): Unit

def logTrace(msg: ⇒ String): Unit

def logWarning(msg: ⇒ String, throwable: Throwable): Unit

def logWarning(msg: ⇒ String): Unit

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

final val outputCols: StringArrayParam

lazy val params: Array[Param[_]]

def save(path: String): Unit

final def set(paramPair: ParamPair[_]): WeightHotEncoderEstimator.this.type

final def set(param: String, value: Any): WeightHotEncoderEstimator.this.type

final def set[T](param: Param[T], value: T): WeightHotEncoderEstimator.this.type

final def setDefault(paramPairs: ParamPair[_]*): WeightHotEncoderEstimator.this.type

final def setDefault[T](param: Param[T], value: T): WeightHotEncoderEstimator.this.type

def setDropLast(value: Boolean): WeightHotEncoderEstimator.this.type

def setGroupCols(value: Array[String]): WeightHotEncoderEstimator.this.type

def setGroupWeighting(value: String): WeightHotEncoderEstimator.this.type