diff --git a/Makefile b/Makefile index 3f55b2ca..414777ad 100644 --- a/Makefile +++ b/Makefile @@ -44,15 +44,15 @@ test: ### Runs all the project tests $(PYTHON) -m pytest tests/ package: clean ### Runs the project setup - echo __version__ = \"$(version)\" > src/ydata_quality/__version__.py + echo __version__ = \"$(version)\" > src/data_quality/__version__.py echo "$(version)" > VERSION $(PYTHON) setup.py sdist bdist_wheel install: ### Installs required dependencies - $(PIP) install dist/ydata-quality-$(version).tar.gz + $(PIP) install dist/data-quality-$(version).tar.gz link-local: ### Installs the lib in dev mode - echo __version__ = \"$(version)\" > src/ydata_quality/__version__.py + echo __version__ = \"$(version)\" > src/data_quality/__version__.py $(PIP) install -e . upload: diff --git a/README.md b/README.md index 4efd6587..6310bf77 100644 --- a/README.md +++ b/README.md @@ -1,33 +1,30 @@ -> **`ydata-quality` is now `data-quality`.** This package has been renamed to `data-quality`. Please follow the [Migration Guide](#migration-guide) as soon as possible — the old package will no longer receive updates or bug fixes. - - -[![](https://api.codacy.com/project/badge/Grade/f1662116e982402c956d2720fbd24507)](https://app.codacy.com/gh/ydataai/ydata-quality) -![](https://img.shields.io/github/workflow/status/ydataai/ydata-quality/release) -![](https://img.shields.io/pypi/status/ydata-quality) -[![](https://pepy.tech/badge/ydata-quality)](https://pypi.org/project/ydata-quality/) +[![](https://api.codacy.com/project/badge/Grade/f1662116e982402c956d2720fbd24507)](https://app.codacy.com/gh/Data-Centric-AI-Community/fg-data-quality) +![](https://img.shields.io/github/workflow/status/Data-Centric-AI-Community/fg-data-quality/release) +![](https://img.shields.io/pypi/status/fg-data-quality) +[![](https://pepy.tech/badge/fg-data-quality)](https://pypi.org/project/fg-data-quality/) ![](https://img.shields.io/badge/python-3.6%20%7C%203.7%20%7C%203.8-blue) -[![](https://img.shields.io/pypi/v/ydata-quality)](https://pypi.org/project/ydata-quality/) -![](https://img.shields.io/github/license/ydataai/ydata-quality) +[![](https://img.shields.io/pypi/v/fg-data-quality)](https://pypi.org/project/fg-data-quality/) +![](https://img.shields.io/github/license/Data-Centric-AI-Community/fg-data-quality) -# YData Quality +# Data Quality -ydata_quality is an open-source python library for assessing Data Quality throughout the multiple stages of a data pipeline development. +data_quality is an open-source python library for assessing Data Quality throughout the multiple stages of a data pipeline development. -A holistic view of the data can only be captured through a look at data from multiple dimensions and `ydata_quality` evaluates it in a modular way wrapped into a single Data Quality engine. This repository contains the core python source scripts and walkthrough tutorials. +A holistic view of the data can only be captured through a look at data from multiple dimensions and `data_quality` evaluates it in a modular way wrapped into a single Data Quality engine. This repository contains the core python source scripts and walkthrough tutorials. ## Quickstart -The source code is currently hosted on GitHub at: https://github.com/ydataai/ydata-quality +The source code is currently hosted on GitHub at: https://github.com/Data-Centric-AI-Community/fg-data-quality -Binary installers for the latest released version are available at the [Python Package Index (PyPI).](https://pypi.org/project/ydata-quality/) +Binary installers for the latest released version are available at the [Python Package Index (PyPI).](https://pypi.org/project/fg-data-quality/) ``` -pip install ydata-quality +pip install fg-data-quality ``` ### Comprehensive quality check in few lines of code ```python -from ydata_quality import DataQuality +from data_quality import DataQuality import pandas as pd #Load in the data @@ -72,7 +69,7 @@ pip uninstall ydata-quality ### 2. Install the new package ```bash -pip install data-quality +pip install fg-data-quality ``` ### 3. Update your imports @@ -97,19 +94,19 @@ grep -r "ydata_quality" . --include="*.py" ## Examples -Here you can find walkthrough tutorials and examples to familiarize with different modules of `ydata_quality` +Here you can find walkthrough tutorials and examples to familiarize with different modules of `data_quality` -- [Start Here for Quick and Overall Walkthrough](https://github.com/ydataai/ydata-quality/blob/master/tutorials/main.ipynb) +- [Start Here for Quick and Overall Walkthrough](https://github.com/Data-Centric-AI-Community/fg-data-quality/blob/master/tutorials/main.ipynb) To dive into any focussed module, and to understand how they work, here are tutorial notebooks: -1. [Bias and Fairness](https://github.com/ydataai/ydata-quality/blob/master/tutorials/bias_fairness.ipynb) -2. [Data Expectations](https://github.com/ydataai/ydata-quality/blob/master/tutorials/data_expectations.ipynb) -3. [Data Relations](https://github.com/ydataai/ydata-quality/blob/master/tutorials/data_relations.ipynb) -4. [Drift Analysis](https://github.com/ydataai/ydata-quality/blob/master/tutorials/drift.ipynb) -5. [Duplicates](https://github.com/ydataai/ydata-quality/blob/master/tutorials/duplicates.ipynb) -6. Labelling: [Categoricals](https://github.com/ydataai/ydata-quality/blob/master/tutorials/labelling_categorical.ipynb) and [Numericals](https://github.com/ydataai/ydata-quality/blob/master/tutorials/labelling_numerical.ipynb) -7. [Missings](https://github.com/ydataai/ydata-quality/blob/master/tutorials/missings.ipynb) -8. [Erroneous Data](https://github.com/ydataai/ydata-quality/blob/master/tutorials/erroneous_data.ipynb) +1. [Bias and Fairness](https://github.com/Data-Centric-AI-Community/fg-data-quality/blob/master/tutorials/bias_fairness.ipynb) +2. [Data Expectations](https://github.com/Data-Centric-AI-Community/fg-data-quality/blob/master/tutorials/data_expectations.ipynb) +3. [Data Relations](https://github.com/Data-Centric-AI-Community/fg-data-quality/blob/master/tutorials/data_relations.ipynb) +4. [Drift Analysis](https://github.com/Data-Centric-AI-Community/fg-data-quality/blob/master/tutorials/drift.ipynb) +5. [Duplicates](https://github.com/Data-Centric-AI-Community/fg-data-quality/blob/master/tutorials/duplicates.ipynb) +6. Labelling: [Categoricals](https://github.com/Data-Centric-AI-Community/fg-data-quality/blob/master/tutorials/labelling_categorical.ipynb) and [Numericals](https://github.com/Data-Centric-AI-Community/fg-data-quality/blob/master/tutorials/labelling_numerical.ipynb) +7. [Missings](https://github.com/Data-Centric-AI-Community/fg-data-quality/blob/master/tutorials/missings.ipynb) +8. [Erroneous Data](https://github.com/Data-Centric-AI-Community/fg-data-quality/blob/master/tutorials/erroneous_data.ipynb) ## Contributing We are open to collaboration! If you want to start contributing you only need to: @@ -123,7 +120,7 @@ You can also join the discussions on our [Discord Community](https://discord.com For support in using this library, please join our Discord server. The Discord community is very friendly and great about quickly answering questions about the use and development of the library. [Click here to join our Discord community!](https://discord.com/invite/mw7xjJ7b7s) ## License -[GNU General Public License v3.0](https://github.com/ydataai/ydata-quality/blob/master/LICENSE) +[GNU General Public License v3.0](https://github.com/Data-Centric-AI-Community/fg-data-quality/blob/master/LICENSE) ## About diff --git a/setup.py b/setup.py index 0bf74e5d..8595fdb4 100644 --- a/setup.py +++ b/setup.py @@ -7,7 +7,7 @@ long_description = (here / 'README.md').read_text(encoding='utf-8') version = (here / 'VERSION').read_text().rstrip("\n") -setup(name='ydata-quality', +setup(name='fg-data-quality', version=version, description='YData open-source tools for Data Quality.', long_description=long_description, @@ -34,8 +34,8 @@ 'Topic :: Software Development :: Libraries :: Python Modules' ], keywords='data science ydata', - url='https://github.com/ydataai/ydata-quality', - license="https://github.com/ydataai/ydata-quality/blob/master/LICENSE", + url='https://github.com/Data-Centric-AI-Community/fg-data-quality', + license="https://github.com/Data-Centric-AI-Community/fg-data-quality/blob/master/LICENSE", python_requires=">=3.7, <3.9", packages=find_namespace_packages('src'), package_dir={'':'src'}, diff --git a/src/data_quality/__init__.py b/src/data_quality/__init__.py new file mode 100644 index 00000000..d850701d --- /dev/null +++ b/src/data_quality/__init__.py @@ -0,0 +1,22 @@ +""" +YData open-source lib for Data Quality. +""" +from warnings import warn + +from .core.data_quality import DataQuality + +from .__version__ import __version__ # noqa: F401 + +warn( + """ + `import ydata_quality` is deprecated and will not receive more updates. + Please install data-quality via `pip install fg-data-quality` and use `import data_quality` instead. + """, + DeprecationWarning, + stacklevel=2, +) + + +__all__ = [ + "DataQuality" +] diff --git a/src/ydata_quality/bias_fairness/__init__.py b/src/data_quality/bias_fairness/__init__.py similarity index 100% rename from src/ydata_quality/bias_fairness/__init__.py rename to src/data_quality/bias_fairness/__init__.py diff --git a/src/ydata_quality/bias_fairness/engine.py b/src/data_quality/bias_fairness/engine.py similarity index 99% rename from src/ydata_quality/bias_fairness/engine.py rename to src/data_quality/bias_fairness/engine.py index 94b099a1..40d0a79f 100644 --- a/src/ydata_quality/bias_fairness/engine.py +++ b/src/data_quality/bias_fairness/engine.py @@ -7,7 +7,7 @@ from pandas import DataFrame, Series from dython.nominal import compute_associations -from src.ydata_quality.core.warnings import Priority +from src.data_quality.core.warnings import Priority from ..core import QualityEngine, QualityWarning from ..utils.correlations import filter_associations diff --git a/src/ydata_quality/core/__init__.py b/src/data_quality/core/__init__.py similarity index 100% rename from src/ydata_quality/core/__init__.py rename to src/data_quality/core/__init__.py diff --git a/src/ydata_quality/core/data_quality.py b/src/data_quality/core/data_quality.py similarity index 100% rename from src/ydata_quality/core/data_quality.py rename to src/data_quality/core/data_quality.py diff --git a/src/ydata_quality/core/engine.py b/src/data_quality/core/engine.py similarity index 100% rename from src/ydata_quality/core/engine.py rename to src/data_quality/core/engine.py diff --git a/src/ydata_quality/core/warnings.py b/src/data_quality/core/warnings.py similarity index 100% rename from src/ydata_quality/core/warnings.py rename to src/data_quality/core/warnings.py diff --git a/src/ydata_quality/data_expectations/__init__.py b/src/data_quality/data_expectations/__init__.py similarity index 100% rename from src/ydata_quality/data_expectations/__init__.py rename to src/data_quality/data_expectations/__init__.py diff --git a/src/ydata_quality/data_expectations/engine.py b/src/data_quality/data_expectations/engine.py similarity index 99% rename from src/ydata_quality/data_expectations/engine.py rename to src/data_quality/data_expectations/engine.py index 04a14736..82ab736d 100644 --- a/src/ydata_quality/data_expectations/engine.py +++ b/src/data_quality/data_expectations/engine.py @@ -6,7 +6,7 @@ from pandas import DataFrame from numpy import argmin -from src.ydata_quality.core.warnings import Priority +from src.data_quality.core.warnings import Priority from ..core import QualityEngine, QualityWarning from ..utils.auxiliary import test_load_json_path diff --git a/src/ydata_quality/data_relations/__init__.py b/src/data_quality/data_relations/__init__.py similarity index 100% rename from src/ydata_quality/data_relations/__init__.py rename to src/data_quality/data_relations/__init__.py diff --git a/src/ydata_quality/data_relations/engine.py b/src/data_quality/data_relations/engine.py similarity index 99% rename from src/ydata_quality/data_relations/engine.py rename to src/data_quality/data_relations/engine.py index 58f8043b..16881daf 100644 --- a/src/ydata_quality/data_relations/engine.py +++ b/src/data_quality/data_relations/engine.py @@ -5,7 +5,7 @@ from numpy import argwhere, ones, tril from pandas import DataFrame -from src.ydata_quality.core.warnings import Priority +from src.data_quality.core.warnings import Priority from ..core import QualityEngine, QualityWarning from ..utils.auxiliary import infer_dtypes, standard_normalize diff --git a/src/ydata_quality/drift/__init__.py b/src/data_quality/drift/__init__.py similarity index 100% rename from src/ydata_quality/drift/__init__.py rename to src/data_quality/drift/__init__.py diff --git a/src/ydata_quality/drift/engine.py b/src/data_quality/drift/engine.py similarity index 100% rename from src/ydata_quality/drift/engine.py rename to src/data_quality/drift/engine.py diff --git a/src/ydata_quality/duplicates/__init__.py b/src/data_quality/duplicates/__init__.py similarity index 100% rename from src/ydata_quality/duplicates/__init__.py rename to src/data_quality/duplicates/__init__.py diff --git a/src/ydata_quality/duplicates/engine.py b/src/data_quality/duplicates/engine.py similarity index 99% rename from src/ydata_quality/duplicates/engine.py rename to src/data_quality/duplicates/engine.py index c12c2ac6..0f23b07b 100644 --- a/src/ydata_quality/duplicates/engine.py +++ b/src/data_quality/duplicates/engine.py @@ -6,7 +6,7 @@ from pandas import DataFrame -from src.ydata_quality.core.warnings import Priority +from src.data_quality.core.warnings import Priority from ..core import QualityEngine, QualityWarning from ..utils.auxiliary import find_duplicate_columns diff --git a/src/ydata_quality/erroneous_data/__init__.py b/src/data_quality/erroneous_data/__init__.py similarity index 100% rename from src/ydata_quality/erroneous_data/__init__.py rename to src/data_quality/erroneous_data/__init__.py diff --git a/src/ydata_quality/erroneous_data/engine.py b/src/data_quality/erroneous_data/engine.py similarity index 99% rename from src/ydata_quality/erroneous_data/engine.py rename to src/data_quality/erroneous_data/engine.py index 7658acf8..f120c28d 100644 --- a/src/ydata_quality/erroneous_data/engine.py +++ b/src/data_quality/erroneous_data/engine.py @@ -6,7 +6,7 @@ from pandas import DataFrame -from src.ydata_quality.core.warnings import Priority +from src.data_quality.core.warnings import Priority from ..core import QualityEngine, QualityWarning from ..utils.enum import DataFrameType diff --git a/src/ydata_quality/labelling/__init__.py b/src/data_quality/labelling/__init__.py similarity index 100% rename from src/ydata_quality/labelling/__init__.py rename to src/data_quality/labelling/__init__.py diff --git a/src/ydata_quality/labelling/engine.py b/src/data_quality/labelling/engine.py similarity index 99% rename from src/ydata_quality/labelling/engine.py rename to src/data_quality/labelling/engine.py index 85b61054..689514d1 100644 --- a/src/ydata_quality/labelling/engine.py +++ b/src/data_quality/labelling/engine.py @@ -5,7 +5,7 @@ from pandas import DataFrame, Series -from src.ydata_quality.core.warnings import Priority +from src.data_quality.core.warnings import Priority from ..core import QualityEngine, QualityWarning from ..utils.auxiliary import infer_dtypes diff --git a/src/ydata_quality/missings/__init__.py b/src/data_quality/missings/__init__.py similarity index 100% rename from src/ydata_quality/missings/__init__.py rename to src/data_quality/missings/__init__.py diff --git a/src/ydata_quality/missings/engine.py b/src/data_quality/missings/engine.py similarity index 99% rename from src/ydata_quality/missings/engine.py rename to src/data_quality/missings/engine.py index b298e7b4..65d21416 100644 --- a/src/ydata_quality/missings/engine.py +++ b/src/data_quality/missings/engine.py @@ -5,7 +5,7 @@ from pandas import DataFrame, Series -from src.ydata_quality.core.warnings import Priority +from src.data_quality.core.warnings import Priority from ..core import QualityEngine, QualityWarning from ..utils.correlations import filter_associations diff --git a/src/ydata_quality/utils/__init__.py b/src/data_quality/utils/__init__.py similarity index 100% rename from src/ydata_quality/utils/__init__.py rename to src/data_quality/utils/__init__.py diff --git a/src/ydata_quality/utils/auxiliary.py b/src/data_quality/utils/auxiliary.py similarity index 100% rename from src/ydata_quality/utils/auxiliary.py rename to src/data_quality/utils/auxiliary.py diff --git a/src/ydata_quality/utils/correlations.py b/src/data_quality/utils/correlations.py similarity index 100% rename from src/ydata_quality/utils/correlations.py rename to src/data_quality/utils/correlations.py diff --git a/src/ydata_quality/utils/enum.py b/src/data_quality/utils/enum.py similarity index 100% rename from src/ydata_quality/utils/enum.py rename to src/data_quality/utils/enum.py diff --git a/src/ydata_quality/utils/logger.py b/src/data_quality/utils/logger.py similarity index 100% rename from src/ydata_quality/utils/logger.py rename to src/data_quality/utils/logger.py diff --git a/src/ydata_quality/utils/modelling.py b/src/data_quality/utils/modelling.py similarity index 100% rename from src/ydata_quality/utils/modelling.py rename to src/data_quality/utils/modelling.py diff --git a/src/ydata_quality/__init__.py b/src/ydata_quality/__init__.py index d850701d..b11eccf3 100644 --- a/src/ydata_quality/__init__.py +++ b/src/ydata_quality/__init__.py @@ -19,4 +19,4 @@ __all__ = [ "DataQuality" -] +] \ No newline at end of file diff --git a/tests/engines/test_data_relations.py b/tests/engines/test_data_relations.py index f87d2870..83e6ab91 100644 --- a/tests/engines/test_data_relations.py +++ b/tests/engines/test_data_relations.py @@ -4,7 +4,7 @@ import nbformat from nbconvert.preprocessors import ExecutePreprocessor -from ydata_quality.data_relations.engine import DataRelationsDetector +from data_quality.data_relations.engine import DataRelationsDetector @fixture(name='data_relations') diff --git a/tutorials/bias_fairness.ipynb b/tutorials/bias_fairness.ipynb index 98b7e318..76ba419e 100644 --- a/tutorials/bias_fairness.ipynb +++ b/tutorials/bias_fairness.ipynb @@ -2,11 +2,12 @@ "cells": [ { "cell_type": "markdown", + "metadata": {}, "source": [ - "# YData Quality - Bias & Fairness Tutorial\n", + "# Data Quality - Bias & Fairness Tutorial\n", "Time-to-Value: 3 minutes\n", "\n", - "This notebook provides a tutorial for the ydata_quality package module on Bias & Fairness .\n", + "This notebook provides a tutorial for the data_quality package module on Bias & Fairness .\n", "\n", "**Structure:**\n", "\n", @@ -16,11 +17,11 @@ "4. Run the quality checks\n", "5. Assess the warnings\n", "6. (Extra) Detailed overview" - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "## Bias and Fairness\n", "As data is increasingly used for automated decision making with heavy impact on individual lifes, a thorough analysis of data quality encompasses an understanding of the inherent biases embedded in the datasets that can cause different treatments based on sensitive attributes. There are ethical and legal obligations for Data Scientists to develop applications which are unbiased and fair.\n", @@ -42,61 +43,33 @@ "- Chapter 11 Bias and Fairness | Big Data and Social Science [(link)](https://textbook.coleridgeinitiative.org/chap-bias.html)\n", "- Fairness and Machine Learning - Limitations and Opportunities [(link)](https://fairmlbook.org/)\n", "- Fairness Tutorial - Moritz Hardt - MLSS 2020, Tübingen [(link)](https://www.youtube.com/watch?v=Igq_S_7IfOU)" - ], - "metadata": {} + ] }, { "cell_type": "code", - "execution_count": 1, + "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "import pandas as pd\n", - "from ydata_quality.bias_fairness import BiasFairness" - ], - "outputs": [], - "metadata": {} + "from data_quality.bias_fairness import BiasFairness" + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "## Load the example dataset\n", "The \"Adult Data Set\" (a.k.a. \"Census Income\") contains a set of records to predict whether an individual's income exceeds $50K/yr, based on census data. " - ], - "metadata": {} + ] }, { "cell_type": "code", "execution_count": 2, - "source": [ - "# This is the DataFrame used in the demo from GE tutorials\n", - "df = pd.read_csv('../datasets/transformed/census_10k.csv')\n", - "df.head()" - ], + "metadata": {}, "outputs": [ { - "output_type": "execute_result", "data": { - "text/plain": [ - " age workclass fnlwgt education education-num marital-status \\\n", - "0 57 Private 132704 Masters 14 Separated \n", - "1 25 Private 321205 Bachelors 13 Never-married \n", - "2 25 Private 38090 Bachelors 13 Never-married \n", - "3 54 Private 29909 11th 7 Married-civ-spouse \n", - "4 43 Private 347934 HS-grad 9 Separated \n", - "\n", - " occupation relationship race sex capital-gain capital-loss \\\n", - "0 Prof-specialty Not-in-family White Male 10520 0 \n", - "1 Exec-managerial Not-in-family White Male 4101 0 \n", - "2 Sales Not-in-family White Male 0 0 \n", - "3 Other-service Wife White Female 0 0 \n", - "4 Sales Not-in-family White Female 0 0 \n", - "\n", - " hours-per-week native-country income workclass2 \n", - "0 32 United-States >50K Private \n", - "1 35 United-States <=50K Private \n", - "2 44 United-States <=50K Private \n", - "3 43 United-States <=50K Private \n", - "4 30 United-States <=50K Private " - ], "text/html": [ "
\n", "