fg-data-profiling README

The project's README from github.com/Data-Centric-AI-Community/fg-data-profiling. As one markdown file: README.md.

fg-data-profiling

ydata-profiling is now fg-data-profiling. This package has been renamed to fg-data-profiling. Please follow the Migration Guide as soon as possible — the old package will no longer receive updates or bug fixes.

Build Status PyPI download month Code Coverage Release Version Python Version Code style: black

Data Profiling Logo

Documentation | Discord | Stack Overflow | Latest changelog

Do you like this project? Show us your love and give feedback!

fg-data-profiling primary goal is to provide a one-line Exploratory Data Analysis (EDA) experience in a consistent and fast solution. Like pandas df.describe() function, that is so handy, fg-data-profiling delivers an extended analysis of a DataFrame while allowing the data analysis to be exported in different formats such as html and json.

The package outputs a simple and digested analysis of a dataset, including time-series and text.

Looking for a scalable solution that can fully integrate with your database systems?
Leverage YData Fabric Data Catalog to connect to different databases and storages (Oracle, snowflake, PostGreSQL, GCS, S3, etc.) and leverage an interactive and guided profiling experience in Fabric. Check out the Community Version.

Migration Guide

1. Uninstall the old package

pip uninstall ydata-profiling

2. Install the new package

pip install fg-data-profiling

3. Update your imports

Find and replace all occurrences of the old import in your codebase:

# Before
import ydata_profiling
from ydata_profiling import ProfileReport

# After
import data_profiling
from data_profiling import ProfileReport

You can use this one-liner to find all affected files:

grep -r "ydata_profiling" . --include="*.py"

▶️ Quickstart

Install

pip install fg-data-profiling

or

conda install -c conda-forge fg-data-profiling

Start profiling

Start by loading your pandas DataFrame as you normally would, e.g. by using:

import numpy as np
import pandas as pd
from data_profiling import ProfileReport

df = pd.DataFrame(np.random.rand(100, 5), columns=["a", "b", "c", "d", "e"])

To generate the standard profiling report, merely run:

profile = ProfileReport(df, title="Profiling Report")

📊 Key features

The report contains three additional sections:

🎁 Latest features

✨ Spark

Spark support has been released, but we are always looking for an extra pair of hands 👐. Check current work in progress!.

📝 Use cases

fg-data-profiling can be used to deliver a variety of different use-case. The documentation includes guides, tips and tricks for tackling them:

Use case Description
Comparing datasets Comparing multiple version of the same dataset
Profiling a Time-Series dataset Generating a report for a time-series dataset with a single line of code
Profiling large datasets Tips on how to prepare data and configure fg-data-profiling for working with large datasets
Handling sensitive data Generating reports which are mindful about sensitive data in the input dataset
Dataset metadata and data dictionaries Complementing the report with dataset details and column-specific data dictionaries
Customizing the report's appearance Changing the appearance of the report's page and of the contained visualizations
Profiling Databases For a seamless profiling experience in your organization's databases, check Fabric Data Catalog, which allows to consume data from different types of storages such as RDBMs (Azure SQL, PostGreSQL, Oracle, etc.) and object storages (Google Cloud Storage, AWS S3, Snowflake, etc.), among others.

Using inside Jupyter Notebooks

There are two interfaces to consume the report inside a Jupyter notebook: through widgets and through an embedded HTML report.

Notebook Widgets

The above is achieved by simply displaying the report as a set of widgets. In a Jupyter Notebook, run:

profile.to_widgets()

The HTML report can be directly embedded in a cell in a similar fashion:

profile.to_notebook_iframe()
HTML

Exporting the report to a file

To generate a HTML report file, save the ProfileReport to an object and use the to_file() function:

profile.to_file("your_report.html")

Alternatively, the report's data can be obtained as a JSON file:

# As a JSON string
json_data = profile.to_json()

# As a file
profile.to_file("your_report.json")

Using in the command line

For standard formatted CSV files (which can be read directly by pandas without additional settings), the data_profiling executable can be used in the command line. The example below generates a report named Example Profiling Report, using a configuration file called default.yaml, in the file report.html by processing a data.csv dataset.

data_profiling --title "Example Profiling Report" --config_file default.yaml data.csv report.html

Additional details on the CLI are available on the documentation.

👀 Examples

The following example reports showcase the potentialities of the package across a wide range of dataset and data types:

🛠️ Installation

Additional details, including information about widget support, are available on the documentation.

Using pip

PyPi Downloads PyPi Monthly Downloads PyPi Version

You can install using the pip package manager by running:

pip install -U fg-data-profiling

Extras

The package declares "extras", sets of additional dependencies.

Install these with e.g.

pip install -U fg-data-profiling[notebook,unicode,pyspark]

Using conda

Conda Downloads Conda Version

You can install using the conda package manager by running:

conda install -c conda-forge fg-data-profiling

From source (development)

Download the source code by cloning the repository or click on Download ZIP to download the latest stable version.

Install it by navigating to the proper directory and running:

pip install -e .

The profiling report is written in HTML and CSS, which means a modern browser is required.

You need Python 3 to run the package. Other dependencies can be found in the requirements files:

Filename Requirements
requirements.txt Package requirements
requirements-dev.txt Requirements for development
requirements-test.txt Requirements for testing
setup.py Requirements for widgets etc.

🔗 Integrations

To maximize its usefulness in real world contexts, fg-data-profiling has a set of implicit and explicit integrations with a variety of other actors in the Data Science ecosystem:

Integration type Description
Other DataFrame libraries How to compute the profiling of data stored in libraries other than pandas
Great Expectations Generating Great Expectations expectations suites directly from a profiling report
Interactive applications Embedding profiling reports in Streamlit, Dash or Panel applications
Pipelines Integration with DAG workflow execution tools like Airflow or Kedro
Cloud services Using fg-data-profiling in hosted computation services like Lambda, Google Cloud or Kaggle
IDEs Using fg-data-profiling directly from integrated development environments such as PyCharm

🙋 Support

Need help? Want to share a perspective? Report a bug? Ideas for collaborations? Reach out via the following channels:

Need Help?
Get your questions answered with a product owner by booking a Pawsome chat! 🐼

❗ Before reporting an issue on GitHub, check out Common Issues.

🤝🏽 Contributing

Learn how to get involved in the Contribution Guide.

A low-threshold place to ask questions or start contributing is the Data Centric AI Community's Discord.

A big thank you to all our amazing contributors!

Contributors wall made with contrib.rocks.

fg-data-profiling (Data-Centric-AI-Community/fg-data-profiling)

Searching Database…

13.7K ·GitHub
Fg-data-profiling
What is this?fg-data-profiling (formerly ydata-profiling / pandas-profiling) is a Python package that generates a one-line exploratory data analysis (EDA) report for pandas or Spark DataFrames, similar to df.describe() but far more detailed.
Is this a library or a standalone application?It's a Python library built around a ProfileReport class; it also ships a small command-line executable called dataprofiling for processing standard CSV files directly from the terminal.
How do I generate a report from my DataFrame?Load your pandas DataFrame as usual, then create a ProfileReport(df) and call its to_file() method to export an HTML report, or get the data as JSON; inside Jupyter it can also render as interactive widgets or an embedded HTML cell.
Do I need Spark or just pandas to use it?Pandas is the default and all you need to get started; Spark support is an optional extra for scaling the analysis to big datasets, enabled via the [pyspark] extra.
Is this the same project as pandas-profiling or ydata-profiling?Yes — it was previously published as pandas-profiling and later ydata-profiling, and has now been renamed to fg-data-profiling. The old packages no longer receive updates or bug fixes, so existing users must follow the migration guide and update their imports.
What's included in the generated report?An Overview with dataset-level stats, an Alerts section flagging issues like high correlation, skewness, missing values and constant columns, plus univariate/multivariate analysis and a Reproduction section with timing and config details.
Can it compare two versions of the same dataset?Yes, it offers a one-line solution to generate a report comparing two datasets, for example data before and after preprocessing.
Does it handle time-series, text, or image data specially?Yes — it adds auto-correlation and seasonality analysis with ACF/PACF plots for time-series columns, script/category detection for text, and file-size, dimension and EXIF checks for images.
What other tools does it integrate with?It integrates with Great Expectations for generating expectation suites, interactive apps like Streamlit, Dash and Panel, pipeline tools like Airflow and Kedro, and cloud environments such as Lambda, Google Cloud and Kaggle.
What are similar projects?great_expectations, cleanlab, sktime, mlcourse.ai, data-science-ipython-notebooks, pygwalker, COVID-19, spark
Which topics is it in?eda, data-quality, spark, text-processing, time-series, jupyter, csv, pandas, data-engineering, big-data, jupyter-notebook, statistics
Creator
Data-Centric-AI-Community
Size
898.3 MB
Stars
13.7K
Updated
CommitSep 11Release4.20.0Sep 11
Age
10y
Commits · 90 Days