Datasets
Models
Downloads: 1
248 lines (247 with data), 25.4 kB
97÷Yäé Ń @ sb d Z d d l m Z m Z m Z m Z d d l m Z d d l m Z d d l
m Z d d l m
Z
d d l m Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l m Z d d
l m Z d d l m Z d d l m Z d d l Z e d
â Z! e d â Z" d Z# d Z$ d g Z% d Z& d Z' d Z( d Z) d Z* Gd d ä d â Z+ d S)zĽIt is an interface for the developed pre-processing functions (factoring and near-zero-variance,
high-linear-correlation) and statistical summaries.
Ú )┌Dict┌List┌TypeVar┌Any)┌ CONSTANTS)┌PyConfigParser)┌ReadersWriters)┌FactoringThread)┌TransformThreadN)┌ OrderedDict)┌feature_selection)┌stats)┌partial┌ DataFramer zMohsen Mesgarpourz-Copyright 2016, https://github.com/mesgarpour┌GPLz1.1zmohsen.mesgarpour@gmail.com┌Releasec
@ sŤ e Z d Z e d d d É ä Z e e e e d d d É ä Z e e e e d d d É ä Z d e e e e e
e d
d d É ä Z e e e e d
d d É ä Z e e e e d
d d É ä Z
d e e e e e
e d
d d É ä Z e e e e d
d d É ä Z e e e e d
d d É ä Z d d e e e e
e e e e g d d d É ä Z d e e e e e e e g d d d É ä Z d e e e e e e e g d d d É ä Z d! d" e e e e e
e e g d# d$ d% É ä Z d& d" e e e e e
e e g d' d( d) É ä Z d* d+ d" e e e e e e
e e g d, d- d. É ä Z e e e e e e
e g d/ d0 d1 É ä Z d2 e e e
e e e d3 d4 d5 É ä Z d S)6┌
PreProcess)┌ output_pathc C s> t j t j â | _ | j j t â | | _ t â | _ d S)zJInitialise the objects and constants.
:param output_path:
N)
┌logging┌ getLoggerr ┌app_name┌_PreProcess__logger┌debug┌__name__┌_PreProcess__output_pathr ┌ _PreProcess__readers_writers)┌selfr ę r ˙HC:\Users\eagle\Documents\GitHub\Analytics_UoW\TCARER\Stats\PreProcess.py┌__init__5 s zPreProcess.__init__)┌df┌includes┌ file_name┌returnc C s | j j d â d } | j j d | j d | d g d d â x┘ | D]Đ } | | k rE | j j d | j d | d d | g d d â t j | | â } t j d
| d d ů d f d | d d ů d
f i â } | j d d d â} | j j d | j d | d | d d d d â qE W| S)a/ Calculate the odds ratio for all the features that are included and all the categorical states.
:param df: the features dataframe.
:param includes: the name of included features.
:param file_name: the name of the summary output file.
:return: the summary output.
z)Produce statistics for discrete features.N┌path┌title┌data┌appendFz Feature NameT┌valuer ┌freqÚ ┌ ascending┌header)
r r r ┌save_csvr r
┌itemfreq┌pdr ┌ sort_values)r r r! r" ┌ summaries┌f_namer r r ┌stats_discrete_df? s*
; z PreProcess.stats_discrete_dfc C s
| j j d â d } | j j d | j d | d g d d â x┼ | D]Ż } | | k rE | j j d | j d | d d | g d d â | | j t j â j d
d d d
d d g â j â } t j
j | â j â } | j j d | j d | d | d d d d â qE W| S)a Calculate the descriptive statistics for all the included continuous features.
:param df: the features dataframe.
:param includes: the name of included features.
:param file_name: the name of the summary output file.
:return: the summary output.
z+Produce statistics for continuous features.Nr$ r% r& r' Fz Feature NameT┌ percentilesgÜÖÖÖÖÖę?g đ?g Ó?g Ŕ?gffffffţ?r, ) r r r r- r ┌applyr/ ┌
to_numeric┌describe┌ transpose┌Series┌to_frame)r r r! r" r1 r2 r r r ┌stats_continuous_df` s*
z PreProcess.stats_continuous_dfF)r ┌categories_dic┌
labels_dic┌
dtypes_dic┌threadedr# c C s | j j d â t | â } | d k r@ | j | | | â } n | j | | | â } g } x. | j â D] } | t | | j â â 7} qh W| j j â | ľ } t j
| d d g âj } x | D]
}
| |
=q├ W| j | d d â} t j | g | d d â} | j
| â } | S)a Categorise groups of features that are selected.
:param df: the features dataframe.
:param categories_dic: the dictionary of the categorical states for the included features.
:param labels_dic: the dictionary of the features names of the categorised features.
:param dtypes_dic: the dictionary of the dtypes of the categorised features.
:param threaded: indicates if it is multi-threaded.
:return: the inputted dataframe with categorised features (if applicable).
z Categorise groups of features.T┌indexr ┌axisr* )r r r ┌(_PreProcess__factoring_group_wise_series┌*_PreProcess__factoring_group_wise_threaded┌keys┌list┌dtypes┌to_dictr/ r ┌drop┌concat┌astype) r r r< r= r> r? ┌pool_df_encoded┌labels_encoded┌ label_group┌
dtype_orig┌labelr r r ┌factoring_group_wiseü s
z PreProcess.factoring_group_wise)r r< r= r# c C sş | j j d â t | | | â } g } y1 x* | j â D] } | j | j | â â q8 WWnM t k
rĘ } z- | j j t d t | â â t
j â WYd d } ~ Xn X| S)ab Categorise a group of features that are selected (single-threaded).
:param df: the features dataframe.
:param categories_dic: the dictionary of the categorical states for the included features.
:param labels_dic: the dictionary of the features names of the categorised features.
:return: the categorised features.
z0Categorise groups of features (single-threaded).z - Invalid configuration(s): N) r r r rD r' ┌factor_arr_multiple┌
ValueError┌errorr ┌str┌sys┌exit)r r r< r= ┌factoring_threadrK rM ┌ exceptionr r r Z __factoring_group_wise_seriesĘ s
z(PreProcess.__factoring_group_wise_seriesc C s┴ | j j d â t | | | â } yK t j d t j â d â Ć( } | j t | j â | j â â } Wd QRXWnM t
k
r╝ } z- | j j t d t
| â â t j â WYd d } ~ Xn X| S)aa Categorise a group of features that are selected (multi-threaded).
:param df: the features dataframe.
:param categories_dic: the dictionary of the categorical states for the included features.
:param labels_dic: the dictionary of the features names of the categorised features.
:return: the categorised features.
z/Categorise groups of features (multi-threaded).┌ processesr* Nz - Invalid configuration(s): )r r r ┌mp┌Pool┌ cpu_count┌mapr rQ rD rR rS r rT rU rV )r r r< r= rW ┌poolrK rX r r r Z __factoring_group_wise_threadedż s
& z*PreProcess.__factoring_group_wise_threadedc
C sˇ | j j d â t | â } | d k r@ | j | | | â } n | j | | | â } t | j â â } | j j â | ľ } t j
| d d g âj } x | D]
} | | =q× W| j | d d â} t j | g | d d â} | j
| â } | S)a■ Categorise features that are selected.
:param df: the features dataframe.
:param categories_dic: the dictionary of the categorical states for the included features.
:param labels_dic: the dictionary of the features names of the categorised features.
:param dtypes_dic: the dictionary of the dtypes of the categorised features.
:param threaded: indicates if it is multi-threaded.
:return: the inputted dataframe with categorised features (if applicable).
z Categorise.Tr@ r rA r* )r r r ┌*_PreProcess__factoring_feature_wise_series┌,_PreProcess__factoring_feature_wise_threadedrE rD rF rG r/ r rH rI rJ )
r r r< r= r> r? rK rL rN rO r r r ┌factoring_feature_wiseË s
z!PreProcess.factoring_feature_wisec C sş | j j d â t | | | â } g } y1 x* | j â D] } | j | j | â â q8 WWnM t k
rĘ } z- | j j t d t | â â t
j â WYd d } ~ Xn X| S)aW Categorise features that are selected (single-threaded).
:param df: the features dataframe.
:param categories_dic: the dictionary of the categorical states for the included features.
:param labels_dic: the dictionary of the features names of the categorised features.
:return: the categorised features.
z Categorise (single-threaded).z - Invalid configuration(s): N) r r r rD r' ┌
factor_arrrR rS r rT rU rV )r r r< r= rW rK rM rX r r r Z __factoring_feature_wise_series° s
z*PreProcess.__factoring_feature_wise_seriesc C s▒ | j j d â t | | | â } y; t j â Ć( } | j t | j â | j â â } Wd QRXWnM t k
rČ } z- | j j
t d t | â â t
j â WYd d } ~ Xn X| S)aV Categorise features that are selected (multi-threaded).
:param df: the features dataframe.
:param categories_dic: the dictionary of the categorical states for the included features.
:param labels_dic: the dictionary of the features names of the categorised features.
:return: the categorised features.
z Categorise (multi-threaded).Nz - Invalid configuration(s): )r r r rZ r[ r] r rb rD rR rS r rT rU rV )r r r< r= rW r^ rK rX r r r Z!__factoring_feature_wise_threaded s
& z,PreProcess.__factoring_feature_wise_threadedN)r ┌excludes┌transform_typer? ┌ method_args┌kwargsr# c
s | j j d â t ł â ë ç f d d ć | j j Dâ } | d k rP t â n | } | j j â } x | D] } d | | <ql Wt j | d d g âj } | j
| â } | d k rÎ | j | | | | Ź \ } } n! | j | | | | | Ź \ } } | | f S) a¤ Transform the included features, using the selected and configured method.
:param df: the features dataframe.
:param excludes: the name of excluded features.
:param transform_type: the transformation type (options: 'scale', 'robust_scale', 'max_abs_scalar',
'normalizer', 'kernel_centerer', 'yeo_johnson', 'box_cox')
:param threaded: indicates if it is multi-threaded.
:param method_args: the transformation arguments, which needs to preserved if it is applied to more than
one data set.
:param kwargs: the input argument for the selected transformation function.
:return: the inputted dataframe with transformed features (if applicable).
zTransform Features.c s" g | ] } | ł k r | Ĺ q Sr r )┌.0rO )rc r r ˙
<listcomp>7 s z+PreProcess.transform_df.<locals>.<listcomp>N┌f8r@ r F)
r ┌info┌set┌columns┌values┌dictrF rG r/ r rJ ┌ _PreProcess__transform_df_series┌"_PreProcess__transform_df_threaded)
r r rc rd r? re rf r! rN rO r )rc r ┌ transform_df# s
!!zPreProcess.transform_df)r r! rd re rf r# c K s | j j d â t | Ź } | d k r1 t â n | } yu| d k rm xb| D] } | j | | | â qM Wn>| d k rá x/| D] } | j | | | â qÇ Wn | d k rË xŘ | D] } | j | | | â q│ Wně | d k rx╔ | D] } | j | | | â qŠ Wną | d k r9xľ | D] } | j | | | â qWnr | d k rlxc | D] } | j | | | â qLWn? | d k rčx0 | D] } | j
| | | â qWn t | â é WnM t k
rű} z- | j j
t d
t | â â t j â WYd d } ~ Xn X| | f S) a| Transform the included features, using the selected and configured method (single-threaded).
:param df: the features dataframe.
:param includes: the name of included features.
:param transform_type: the transformation type (options: 'scale', 'robust_scale', 'max_abs_scalar',
'normalizer', 'kernel_centerer', 'yeo_johnson', 'box_cox')
:param method_args: the transformation arguments, which needs to preserved if it is applied to more than
one data set.
:param kwargs: the input argument for the selected transformation function.
:return: the transformed feature.
z%Transform features (single-threaded).N┌scale┌ robust_scale┌max_abs_scalar┌
normalizer┌kernel_centerer┌ yeo_johnson┌box_coxz - Invalid configuration(s): )r r r
rn ┌transform_scale_arr┌transform_robust_scale_arr┌ transform_max_abs_scalar_arr┌transform_normalizer_arr┌ transform_kernel_centerer_arr┌transform_yeo_johnson_arr┌transform_box_cox_arr┌ ExceptionrR rS r rT rU rV ) r r r! rd re rf ┌transform_thread┌namerX r r r Z__transform_df_seriesH s<
z PreProcess.__transform_df_seriesc
K sq | j j d â t j â } | j t t | | j | | j j j
â â â â } t | Ź } | d k rr t â n | } yxt j d t j
â d â ĆU} | d k r╚ | j t | j | | â | â n | d k r÷ | j t | j | | â | â n˛ | d k r$| j t | j | | â | â n─ | d k rR| j t | j | | â | â nľ | d k rÇ| j t | j | | â | â nh | d
k r«| j t | j | | â | â n: | d k r▄| j t | j | | â | â n t | â é Wd QRXWnM t k
r?}
z- | j j t d t |
â â t j â WYd d }
~
Xn Xx$ | j â D] \ } } | | | <qMW| | f S)
aV Transform the included features, using the selected and configured method (multi-threaded).
:param df: the features dataframe.
:param includes: the name of included features.
:param transform_type: the transformation arguments, which needs to preserved if it is applied to more than
one data set.
:param method_args: the transformation arguments, which needs to preserved if it is applied to more than
one data set.
:param kwargs: the input argument for the selected transformation function.
:return: the transformed feature.
z$Transform features (multi-threaded).NrY r* rr rs rt ru rv rw rx z - Invalid configuration(s): ) r r rZ ┌Managerrn rE ┌ziprl ┌Trm ┌tolistr
r[ r\ r] r ry rz r{ r| r} r~ r rÇ rR rS r rT rU rV ┌items)
r r r! rd re rf ┌manager┌dtrü r^ rX ┌k┌vr r r Z__transform_df_threadedz s8 5 " " " " " " " z"PreProcess.__transform_df_threadedgffffffţ?T)r rc r" ┌thresh_corr_cut┌ to_searchr# c
s | j j d â d } | ł } t ł â ë g } t â } | d k rk| ç f d d ć | j Dâ j d d â } x§ | j j D]š }
t | t | |
â | k j â } t
| â d k rÇ y | j |
â Wn t o┘ t
k
rŠ Yn Xt j | | â } x- | D]% } | | j â k r | j | â q Wt
| â d k rÇ | | |
<| j j d
|
d t | â â qÇ W| j | | | t j j | j | d â â } x ł D] }
| |
| |
<qáWt t j | j â â rÔ| j d
d â } | d k r| | d <| | d <| | f S)aś Find and optionally remove the selected highly linearly correlated features.
The Pearson correlation coefficient was calculated for all the pair of variables to measure linear dependence
between them.
:param df: the features dataframe.
:param excludes: the name of excluded features.
:param file_name: the name of the summary output file.
:param thresh_corr_cut: the numeric value for the pair-wise absolute correlation cutoff. e.g. 0.95.
:param to_search: to search or use the saved configuration.
:return: the inputted dataframe with exclusion of features that were selected to be removed.
z=Remove features with high linear correlation (if applicable).NTc s" g | ] } | ł k r | Ĺ q Sr r )rg ┌col)rc r r rh Ă s z9PreProcess.high_linear_correlation_df.<locals>.<listcomp>┌method┌pearsonr* r zHigh Linear Correlation: z ~ z.inirH zFeatures MatcheszCorrelation Matrix)r r rk r rl ┌corrrm rE ┌absr@ ┌len┌removerR ┌AttributeError┌np┌union1drD rj rT ┌_PreProcess__remove┌osr$ ┌joinr ┌any┌isnan┌ reset_index)r r rc r" rî rŹ rĹ ┌ df_excludes┌matchesr1 rO Z matches_temp┌matchré r )rc r ┌high_linear_correlation_dfş s>
,#
&.
z%PreProcess.high_linear_correlation_dfgÜÖÖÖÖÖę?)r rc r" ┌thresh_variancerŹ r# c sü | j j d â | | } t | â } g } t â ë t â } x- | j j â D] } | ł | j j | â <qN W| d k rŮ t j | â }
|
j d d â } ç f d d ć | Dâ } x' | D] }
|
| k r╗ | |
g 7} q╗ W| j
| d t | â i | t j
j | j | d â â } x | D] } | | | | <q Wt t j | j â â ra| j d d â } | d k rw| | d <| | f S)
aë Find and optionally remove the selected near-zero-variance features (Scikit algorithm).
Feature selector that removes all low-variance features.
This feature selection algorithm looks only at the features (X), not the desired outputs (y), and can thus be
used for unsupervised learning.
:param df: the features dataframe.
:param excludes: the name of excluded features.
:param file_name: the name of the summary output file.
:param thresh_variance: Features with a training-set variance lower than this threshold will be removed.
The default is to keep all features with non-zero variance, i.e. remove the features that have the same
value in all samples.
:param to_search: to search or use the saved configuration.
:return: the inputted dataframe with exclusion of features that were selected to be removed.
zPRemove features with near-zero-variance (if applicable), using Scikit algorithm.T┌indicesc s g | ] } ł | Ĺ q Sr r )rg r@ )rú r r rh s z7PreProcess.near_zero_var_df_sklearn.<locals>.<listcomp>┌NZVz.inirH zFeatures Matches)r r rk r rl rm ┌get_locr ZVarianceThresholdZ get_supportrś rE rÖ r$ rÜ r rŤ rľ rť r@ rŁ )r r rc r" ró rŹ r× rč r1 rO Z
variances_Zmatches_indicesZmatches_labelsrá ré r )rú r ┌near_zero_var_df_sklearnŠ s.
:
z#PreProcess.near_zero_var_df_sklearnÚd iŔ )r rc r" ┌thresh_unique_cut┌thresh_freq_cutrŹ r# c
C s¬ | j j d â | | } t | â } g } t â } | d k rx├ | j j D]Á }
t | |
j d t t j t
t j
f â sŁ t j t j | |
â â rş | |
g 7} qN | j
| |
|
| | | â \ } | |
<| d k rN | |
g 7} | j j d |
â qN W| j | d t | â i | t j j | j | d â â } x | D] } | | | | <qHWt t j | j â â rŐ| j d d â } | d k rá| | d <| | f S) a╬ Find and optionally remove the selected near-zero-variance features (custom algorithm).
The features that had constant counts less than or equal a threshold may be filtered out,
to exclude highly constants and near-zero variances.
Rules are as the following:
- Frequency ratio: The frequency of the most prevalent value over the second most frequent value to be
greater than a threshold;
- Percent of unique values: The number of unique values divided by the total number of samples to be greater
than the threshold.
:param df: the features dataframe.
:param excludes: the name of excluded features.
:param file_name: the name of the summary output file.
:param thresh_unique_cut: the cutoff for the percentage of distinct values out of the number of total samples
(upper limit). e.g. 10 * 100 / 100.
:param thresh_freq_cut: the cutoff for the ratio of the most common value to the second most common value
(lower limit). e.g. 95/5.
:param to_search: to search or use the saved configuration.
:return: the inputted dataframe with exclusion of features that were selected to be removed.
zPRemove features with near-zero-variance (if applicable), using custom algorithm.Tr zNear Zero Variance: rĄ z.inirH zFeatures Matches)r r rk r rl rm ┌
isinstance┌iloc┌intrľ ┌floatrť ┌sum┌_PreProcess__near_zero_varrj rś rE rÖ r$ rÜ r rŤ r@ rŁ )
r r rc r" rĘ rę rŹ r× rč r1 rO rá ré r r r ┌near_zero_var_df s0
-
#
:
zPreProcess.near_zero_var_df)┌arrrO rc rĘ rę r# c C s7 | j j d â t j | d d â\ } } t | â d k r_ d d t | â d t | â i f St | d d â} | | k r┼ t | â d t t | â â | k r┼ d d t | â d t | â i f S| | k r| d t | d â | k rd d t | â d t | â i f Sd
d t | â d t | â i f Sd S) a8 Assess a single feature for near-zero-variance (custom algorithm).
The features that had constant counts less than or equal a threshold may be filtered out,
to exclude highly constants and near-zero variances.
Rules are as the following:
- Frequency ratio: The frequency of the most prevalent value over the second most frequent value to be
greater than a threshold;
- Percent of unique values: The number of unique values divided by the total number of samples to be greater
than the threshold.
:param arr: the feature value.
:param label: the feature name.
:param excludes: the name of excluded features.
:param thresh_unique_cut: the cutoff for the percentage of distinct values out of the number of total samples
(upper limit). e.g. 10 * 100 / 100.
:param thresh_freq_cut: the cutoff for the ratio of the most common value to the second most common value
(lower limit). e.g. 95/5.
:return: indicates if the feature has near-zero-variance.
z@Find near-zero-variance (if applicable), using custom algorithm.┌
return_countsTr* ┌unique┌counts┌reverserž r FN)r r rľ r│ rô rE ┌sortedrş )r r▒ rO rc rĘ rę r│ r┤ r r r Z__near_zero_varT s "2"*"zPreProcess.__near_zero_var┌features)r ┌ dict_matchesrŹ r$ ┌sectionr# c s° | j j d â t | t j â } | d k rx | j â | j | | â | j j d | â } | d k rx | j â ł S| j â | j
| â } | j j d d j | â â ç f d d ć | j â Dâ } t
| â d k r˘ ł j | d
d âë ł S) aŽ Confirm removals and if confirmed, then re-read the selected features, then remove
:param df: the features dataframe.
:param dict_matches: the matched features.
:param to_search: to search or use the saved configuration.
:param path: the file path to the configuration file.
:param section: the section name in the configuration file.
:return: the updated features.
z/Confirm removals and implement removal process.Tz:the features defined in the following file to be removed: FzThe feature removal list: ˙,c s/ g | ]% } | D] } | ł k r | Ĺ q q Sr r )rg rM rO )r r r rh Ť s z'PreProcess.__remove.<locals>.<listcomp>r rA r* )r r r r r ┌reset┌
write_dictr ┌question_overwrite┌refresh┌ read_dictrÜ rm rô rH ) r r rŞ rŹ r$ r╣ ┌config┌response┌labelsr )r r Z__removey s"
zPreProcess.__remove) r ┌
__module__┌ __qualname__rT r ┌PandasDataFramer r3 r; r ┌boolrP rB rC ra r_ r` r rq ro rp rş ┌CollectionsOrderedDictrí rŽ r░ rk r» rś r r r r r 4 sR
#'#'! # 1 2-8-206 %r ),┌__doc__┌typingr r r r ┌Configs.CONSTANTSr ┌ ReadersWriters.PyConfigParserr ┌ ReadersWriters.ReadersWritersr ZStats.FactoringThreadr ZStats.TransformThreadr
rÖ rU ┌numpyrľ ┌pandasr/ ┌multiprocessingrZ ┌ collectionsr Zsklearnr ┌ scipy.statsr
┌ functoolsr r r┼ ră ┌
__author__┌
__copyright__┌ __credits__┌ __license__┌ __version__┌__maintainer__┌ __email__┌
__status__r r r r r ┌<module> s6 "