Datasets
Models
Downloads: 1
212 lines (211 with data), 19.2 kB
■ă§YP Ń @ s d Z d d l m Z m Z m Z m Z m Z d d l Z d d l Z d d l Z d d l
m Z d d l m
Z
d d l m Z d d l m Z d d l m Z d d l m Z e d
â Z d Z d Z d g Z d
Z d Z d Z d Z d Z Gd d ä d â Z d S)z; It is an interface for the developed readers and writers.
Ú )┌List┌TypeVar┌Dict┌Any┌CallableN)┌CsvFile)┌ MysqlCommand)┌MysqlConnection)┌PickleSerialised)┌TextFile)┌ MysqlQueries┌ DataFramezMohsen Mesgarpourz-Copyright 2016, https://github.com/mesgarpour┌GPLz1.1zmohsen.mesgarpour@gmail.com┌Releasec @ s§ e Z d Z e e e e e d d d É ä â Z e e e e d d d É ä â Z e d e e e e d d d É ä â Z e e e e e d d
d É ä â Z e e e e e d d
d É ä â Z
e e e e e e d d d É ä â Z e e e e e d d d É ä â Z
e e e e d d d É ä â Z e e e e d d d É ä â Z e d e e e e d d d É ä â Z e e e e e d d d É ä â Z e e e e d d d É ä â Z e e e e d d d! É ä â Z e d" e e e e g d" f e e e d# d$ d% É ä â Z e d& d e e e e e e g d" f e e e d' d( d) É ä â Z e d& d* e e e e e e g d" f d+ d, d- É ä â Z e e e e d. d/ d0 É ä â Z e e e e d. d1 d2 É ä â Z e d3 e e e d4 d5 d6 É ä â Z e d3 d" d" d" e e e e e e e e e g d" f d7 d8 d9 É ä â Z e d: d3 d e e e e e e e e e g d" f d; d< d= É ä â Z e e e e e e d> d? d@ É ä â Z! e e e e e dA dB dC É ä â Z" e e e dD dE dF É ä â Z# e d e e e dG dH dI É ä â Z$ e d* e e e dG dJ dK É ä â Z% e e& â e& â dL e e e e e e e e dM dN dO É ä â Z' e e e e e dP dQ dR É ä â Z( e e e dS dT dU É ä â Z) e e e dV dW dX É ä â Z* e e dY dZ d[ É ä â Z+ d" S)\┌ReadersWriters)┌path┌title┌ext┌returnc C s& t â } | j | | | â | j â S)z°Check if the serialised object exists.
:param path: the output directory path.
:param title: the title of the output file.
:param ext: the extension of the output file.
:return: indicates if the file exists.
)r
┌set┌exists)r r r ┌readerę r ˙UC:\Users\eagle\Documents\GitHub\Analytics_UoW\TCARER\ReadersWriters\ReadersWriters.py┌exists_serialised- s
z ReadersWriters.exists_serialised)┌ db_schema┌db_tabler c C sL t â } | j | â } t j | | d d â} | d k oK t | â d k S)z│Check if MySQL table exists.
:param db_schema: the MySQL database schema.
:param db_table: the MySQL table.
:return: indicates if the file exists.
┌ dataframingTNr )r ┌ exists_tabler ┌load_mysql_query┌len)r r ┌queries┌query┌outputr r r ┌ exists_mysql; s zReadersWriters.exists_mysql┌csvc C s& t â } | j | | | â | j â S)a Check if the CSV file exists.
:param path: the directory path of the CSV file.
:param title: the file name of the CSV file.
:param ext: the extension of the CSV file (default: 'csv').
:return: indicates if the file exists.
)r r r )r r r r r r r ┌
exists_csvH s
zReadersWriters.exists_csvc C s& t â } | j | | | â | j â S)z¨Check if the text file exists.
:param path: the directory path of the text file.
:param title: the file name of the text file.
:param ext: he extension of the text file.
:return: indicates if the file exists.
)r r r )r r r r r r r ┌ exists_textV s
zReadersWriters.exists_text)r r ┌columnr c C sO t â } | j | | â } t j | | d d â} | d k oN t | â d k S)zţCheck if a column exists in a MySQL table.
:param db_schema: the MySQL database schema.
:param db_table: the MySQL table.
:param column: name of the column.
:return: indicates if the column exists.
r TNr )r ┌
exists_columnr r r )r r r( r! r" r# r r r ┌exists_mysql_columnd s
z"ReadersWriters.exists_mysql_column)r r r( r r c C s) t â } | j | | | â | j | â S)a4 Check if the CSV file exists.
:param path: the directory path of the CSV file.
:param title: the file name of the CSV file.
:param column: name of the column.
:param ext: the extension of the CSV file (default: 'csv').
:return: indicates if the column exists.
)r r r) )r r r( r r r r r ┌exists_csv_columns s z ReadersWriters.exists_csv_columnc C s& t â } | j | | | â | j â S)a Check the size of the saved file.
:param path: the directory path of the serialised file.
:param title: the title of the output file.
:param ext: the extension of the output file.
:return: showing stat information of the file.
)r
r ┌size)r r r r r r r ┌size_serialisedâ s
z ReadersWriters.size_serialisedc C s: t j | | d d â} t | d d â r6 | d k Sd S)z┐Check number of records in a MySQL table.
:param db_schema: the MySQL database schema.
:param db_table: the MySQL table.
:return: number of rows in the table.
r Tr N)r ┌load_mysql_table┌int)r r r# r r r ┌size_mysql_tableĹ s z ReadersWriters.size_mysql_table)r" r r c C s: t j | | d d â} t | d d â r6 | d k Sd S)zňCheck number of records in the output result of the MySQL query.
:param query: the mysql query to execute.
:param db_schema: the MySQL database schema.
:return: number of rows in the output table.
r Tr N)r r r/ )r" r r# r r r ┌size_mysql_queryť s z ReadersWriters.size_mysql_queryc C s& t â } | j | | | â | j â S)z■Check number of lines in the CSV file.
:param path: the directory path of the CSV file.
:param title: the title of the CSV file.
:param ext: the extension of the CSV file.
:return: number of lines in the CSV file.
)r r r, )r r r r r r r ┌size_csvž s
zReadersWriters.size_csvc C s& t â } | j | | | â | j â S)a Check number of lines in the text file.
:param path: the directory path of the text file.
:param title: the title of the text file.
:param ext: the extension of the text file.
:return: number of lines in the text file.
)r r r, )r r r r r r r ┌ size_textÁ s
zReadersWriters.size_text)r r ┌objectsc C s- t â } | j | | d â | j | â d S)zŰSerialise the object (Pickle protocol=4), without compression.
:param path: the directory path of the serialised file.
:param title: the title of the output file.
:param objects: the object to be saved.
┌pickleN)r
r ┌save)r r r4 ┌writerr r r ┌save_serialised├ s z ReadersWriters.save_serialisedc C s- t â } | j | | d â | j | â d S)z¸Serialise the object (Pickle protocol=4), then compress (BZ2 compression).
:param path: the directory path of the serialised file.
:param title: the title of the output file.
:param objects: the object to be saved.
┌bz2N)r
r ┌save_bz2)r r r4 r7 r r r ┌save_serialised_compressedđ s z)ReadersWriters.save_serialised_compressedN)r" ┌datar r ┌batchc
C s t â } | j | â | j â } t | | j â } | d k rP | j | â n« | d k r{ | j | | | | d d ânâ t t j t
| â | â â } xa t d | â D]P } | j | | | | | d | ů | | d t | | | d | g â âq¬ W| j
â d S)aŹ Write several rows of data into a MySQL table, using dataframe or list.
:param query: the mysql query to execute, if data type is list.
:param data: the mysql query to execute (applicable for data type of list).
:param db_schema: the MySQL database schema.
:param db_table: the MySQL table.
:param batch: indicates if data to be written in batches.
N┌ batch_title┌allr Ú )r r ┌openr ┌db_session_vars┌write┌
write_manyr/ ┌math┌ceilr ┌range┌str┌close)
r" r< r r r= ┌db┌engine┌dbcr, ┌ir r r ┌
save_mysqlŢ s
+#zReadersWriters.save_mysqlF)r r r< ┌appendr ┌kwargsc K sF t â } | j | | | â | d k r2 | j â | j | | Ź d S)aĚ Append to CSV file using dataframe, dictionary or list.
:param path: the directory path of the CSV file.
:param title: the file name of the CSV file.
:param data: the data to write.
:param append: indicates if data to be appended to an existing file.
:param ext: the extension of the CSV file (default: 'csv').
:param kwargs: any other arguments that the selected writer may accept.
FN)r r ┌resetrO )r r r< rO r rP r7 r r r ┌save_csv˙ s
zReadersWriters.save_csv┌log)r r r< c C sC t â } | j | | | â | d k r2 | j â | j | â d S)aY Append to text file using dataframe, dictionary or list.
:param path: the directory path of the text file.
:param title: the file name of the text file.
:param data: the data to write.
:param append: indicates if data to be appended to an existing file.
:param ext: the extension of the CSV file.
FN)r r rQ rO )r r r< rO r r7 r r r ┌ save_text s
zReadersWriters.save_text)r r r c C s& t â } | j | | d â | j â S)z┌Load a serialised object, that was not compressed.
:param path: the directory path of the serialised file.
:param title: the title of the output file.
:return: the loaded python object.
r5 )r
r ┌load)r r r r r r ┌load_serialised" s z ReadersWriters.load_serialisedc C s& t â } | j | | d â | j â S)zŔLoad a serialised object, that was compressed (BZ2 compression).
:param path: the directory path of the serialised file.
:param title: the title of the output file.
:return: the loaded python object.
r9 )r
r ┌load_bz2)r r r r r r ┌load_serialised_compressed. s z)ReadersWriters.load_serialised_compressedT)r r r c C s d | } t j | | | â S)a7 "Read a MySQL table and return it as a dataframe or a list.
:param db_schema: the MySQL database schema.
:param db_table: the MySQL table.
:param dataframing: indicates if the return data will be dataframe or list.
:return: the output of the executed query (read table).
zSELECT * FROM )r r )r r r r" r r r r. : s
z ReadersWriters.load_mysql_table)r" r r r= ┌float_round_vars┌ float_roundr c
C s] t â } | j | â | j â } t | | j â } | j | | | | | â } | j â | S)a* Execute a MySQL query and return the output result as dataframe or list.
:param query: the mysql query to execute.
:param db_schema: the MySQL database schema.
:param dataframing: indicates if the return data will be dataframe or list.
:param batch: indicates if data is loaded batch by batch.
:param float_round_vars: list of float variables that needs to be rounded.
:param float_round: the rounding precision for the 'float_round_vars' option.
:return: the output of the executed query.
)r r rA r rB ┌readrI )
r" r r r= rY rZ rJ rK rL r# r r r r G s
z ReadersWriters.load_mysql_queryr )r r ┌skipr r rP r c K s/ t â } | j | | | â | j | | | Ź S)aţ Read the CSV file into dataframe or list.
:param path: the directory path of the CSV file.
:param title: the file name of the CSV file.
:param skip: lines to skip before reading or writing.
:param dataframing: indicates if the outputs must be saved into dataframe.
:param ext: the extension of the CSV file (default: 'csv').
:param kwargs: any other arguments that the selected reader may accept.
:return: the read file contents.
)r r r[ )r r r\ r r rP r r r r ┌load_csv_ s zReadersWriters.load_csv)r r r r\ r c C s) t â } | j | | | â | j | â S)a Read the text file into dataframe or list.
:param path: the directory path of the text file.
:param title: the file name of the text file.
:param ext: the extension of the text file.
:param skip: lines to skip before reading or writing.
)r r r[ )r r r r\ r r r r ┌ load_texts s zReadersWriters.load_text)┌ db_procedure┌argsr r c C sT t â } | j | â | j â } t | | j â } | j | | â } | j â | S)a7 Execute a MySQL procedure, and return the raw output results (if applicable).
:param db_procedure: the mysql procedure to execute.
:param args: the procedure's input arguments.
:param db_schema: the MySQL database schema.
:return: the raw output results (if applicable).
)r r rA r rB ┌ call_procrI )r_ r` r rJ rK rL r# r r r ┌load_mysql_procedureé s
z#ReadersWriters.load_mysql_procedure)r r c C s d | } t j | | â d S)züTruncate the MySQL table.
:param db_schema: the MySQL database schema.
:param db_table: the MySQL table.
zTRUNCATE TABLE N)r r )r r r" r r r ┌reset_mysql_tableö s
z ReadersWriters.reset_mysql_table)r r r c C s* t â } | j | | | â | j â d S)z▄Reset the CSV file reader/writer.
:param path: the directory path of the CSV file.
:param title: the file name of the CSV file.
:param ext: the extension of the CSV file (default: 'csv').
N)r r rQ )r r r r r r r ┌ reset_csv× s zReadersWriters.reset_csvc C s* t â } | j | | | â | j â d S)z¤Reset the text file reader/writer.
:param path: the directory path of the text file.
:param title: the file name of the text file.
:param ext: the extension of the text file.
N)r r rQ )r r r r r r r ┌
reset_textź s zReadersWriters.reset_textz$ENGINE=InnoDB DEFAULT CHARSET=latin1)┌table┌columns┌ col_types┌defaults┌ primary_keys┌ unique_keys┌metar c C s( t â } | j | | | | | | | â S)aň Generate a bespoke create query.
:param table: the MySQL table.
:param columns: the list of columns.
:param col_types: the list of column types for columns.
:param defaults: the list of default values for columns.
:param primary_keys: the list of primary keys for the table.
:param unique_keys: the list of unique keys for the table.
:param meta: the table metadata (Engine and charset).
:return: the MySQL query
)r ┌create)rf rg rh ri rj rk rl r! r r r ┌mysql_query_createŞ s z!ReadersWriters.mysql_query_create)rf rg rh r c C s t â } | j | | | â S)zŮGenerate a bespoke insert query.
:param table: the MySQL table.
:param columns: the list of columns.
:param col_types: the list of column types for columns.
:return: the MySQL query
)r ┌insert)rf rg rh r! r r r ┌mysql_query_insert═ s
z!ReadersWriters.mysql_query_insert)rf r c C s t â } | j | â S)zoGenerate a bespoke drop query.
:param table: the MySQL table.
:return: the MySQL query
)r ┌drop)rf r! r r r ┌mysql_query_drop┌ s z ReadersWriters.mysql_query_drop)┌namer c C sc x\ t d | d â j â } | d k r7 t d â d S| d k rQ t d â d St d â q Wd
S) z▒Commandline question, which aims to get confirmation of the user.
:param name: the subject name, which needs confirmation.
:return: the user's response.
zConfirm or reject z,
>> Print 'y' to accept or 'n' to decline: ┌y┌ApprovedT┌n┌DeclinedFz!ERROR: Invalid command 'y' or 'n'N)┌input┌lower┌print)rs ┌responser r r ┌question_overwriteŃ s
z!ReadersWriters.question_overwrite)r c C sć t j j | â sé y t j | d d âWnV t k
rü } z6 t j t d j | j d t
| j d â â â WYd d } ~ Xn Xd S)zuCreate directories in the specified path if do not exist.
:param path: the directory path to a file.
┌exist_okTz* - Directory creation error: {0:d}:
{1:s}r r@ N) ┌osr r ┌makedirs┌OSError┌sys┌exit┌__name__┌formatr` rH )r ┌er r r ┌create_directories˘ s
z!ReadersWriters.create_directories),râ ┌
__module__┌ __qualname__┌ staticmethodrH ┌boolr r$ r& r' r* r+ r/ r- r0 r1 r2 r3 r r8 r; r r ┌PandasDataFramerN r rR rT rV rX r. r r] r^ rb rc rd re ┌listrn rp rr r| rć r r r r r , s╬ $ $
7%4
!
=+ !
!
0
r ) ┌__doc__┌typingr r r r r rE r~ rü ┌ReadersWriters._CsvFiler ┌ ReadersWriters._MysqlCommandr ┌ ReadersWriters._MysqlConnectionr ┌ ReadersWriters._PickleSerialisedr
┌ReadersWriters._TextFiler ┌ ReadersWriters._MysqlQueriesr rő ┌
__author__┌
__copyright__┌ __credits__┌ __license__┌ __version__┌__maintainer__┌ __email__┌
__status__r r r r r ┌<module> s( (