Global

data-processing
28 projects tagged “data-processing” that gained the most stars yesterday
#599 · +63 · 1d ·GitHub

frame objects, statistical functions, and much more

Creator
pandas-dev
Size
410.4 MB
Stars
50.2K
Age
16y
Commits · 90 Days
#1195 · +27 · 1d ·GitHub

Scrapy, a fast high-level web crawling & scraping framework for Python.

Creator
scrapy
Size
32.8 MB
Stars
64.9K
Age
16y
Commits · 90 Days
#1205 · +27 · 1d ·GitHub

fetchuser(800333171, replaceiconnamewithurl=True) to get the parsed data with asset urls.

Creator
MetaCubeX
Size
13.5 MB
Stars
35K
Age
5y
Commits · 90 Days
#2530 · +9 · 1d ·GitHub

Polars is an analytical query engine for DataFrames, written in Rust. It is designed to be fast, easy to use and expressive. Key features are:

Creator
pola-rs
Size
228.1 MB
Stars
40.1K
Age
6y
Commits · 90 Days
#3262 · +6 · 1d ·GitHub

Incremental engine for long horizon agents 🌟 Star if you like it!

Size
115.9 MB
Stars
11.7K
Age
1y
Commits · 90 Days
#3768 · +4 · 1d ·GitHub

Apache Flink is an open source stream processing framework with powerful stream- and batch-processing capabilities.

Creator
apache
Size
543.8 MB
Stars
26.4K
Updated
CommitOct 19
Age
12y
Commits · 90 Days
#3869 · +4 · 1d ·GitHub

Apache Arrow is the universal columnar format and multi-language toolbox for fast data interchange and in-memory analytics

Creator
apache
Size
231.2 MB
Stars
17.2K
Age
10y
Commits · 90 Days
#4059 · +4 · 1d ·GitHub

A substantial amount of effort has been put into making sure the outputs match exactly with what you would get with HuggingFace Tokenizers in this setting, but this is at a non-negligible cost to performance.

Creator
marcelroed
Size
2.9 MB
Stars
4.1K
Updated
CommitAug 06
Age
11mo
Commits · 90 Days
#4188 · +3 · 1d ·GitHub

IMPORTANT] > ### Thank you Clippy! > > ### But our Sheet is in another Workbook! > > The new source repository URL is .

Creator
SheetJS
Size
103.3 MB
Stars
36.3K
Updated
CommitFeb 01
Age
13y
Commits · 90 Days
#4500 · +3 · 1d ·GitHub

PRQL is a modern language for transforming data — a simple, powerful, pipelined SQL replacement

Creator
PRQL
Size
25.8 MB
Stars
10.9K
Age
4y
Commits · 90 Days
#4807 · +2 · 1d ·GitHub

uv is a fast Python package installer and resolver. To get started:

Creator
wesm
Size
63.7 MB
Stars
25K
Updated
CommitOct 17
Age
14y
Commits · 90 Days
#5467 · +1 · 1d ·GitHub

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

Creator
pathwaycom
Size
178.7 MB
Stars
62.1K
Age
3y
Commits · 90 Days
#5811 · +1 · 1d ·GitHub

Apache Hadoop

Creator
apache
Size
598.2 MB
Stars
15.7K
Updated
CommitOct 19
Age
12y
Commits · 90 Days
#6151 · +1 · 1d ·GitHub

This blog will focus on simple bash commands for parsing data and Linux system maintenance that i acquired from work and LPIC exam. I apologize that there are no detailed citation for all the commands, but they are probably from dear search engine and Stack Overflow.

Creator
onceupon
Size
1.1 MB
Stars
10.8K
Updated
CommitSep 22
Age
10y
Commits · 90 Days
#6202 · +1 · 1d ·GitHub

Modin is a drop-in replacement for pandas. While pandas is single-threaded, Modin lets you instantly speed up your workflows by scaling pandas so it uses all of your cores. Modin works especially well on larger datasets, where pandas becomes painfully slow or runs out of memory.

Size
52.6 MB
Stars
10.4K
Age
8y
Commits · 90 Days
#6239 · +1 · 1d ·GitHub

Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON

Creator
johnkerl
Size
212.3 MB
Stars
10K
Age
11y
Commits · 90 Days
#6828 · +0 · 1d ·GitHub
archived

Due to lack of ability to find an invested maintainer and inability to find time to do routine maintenance and community building, this package is no longer maintained. Please see the discussion 🤝 Maintainer help wanted for more information.

Size
2.4 MB
Stars
20.8K
Age
12y
Commits · 90 Days
#6892 · +0 · 1d ·GitHub

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

Creator
spotify
Size
11.3 MB
Stars
18.8K
Age
14y
Commits · 90 Days
#6948 · +0 · 1d ·GitHub

High-performance AI pipeline engine with a C++ core and 50+ Python-extensible nodes. Build, debug, and scale LLM workflows with 13+ model providers, 8+ vector databases, and agent orchestration, all from your IDE.

Size
127.1 MB
Stars
17.7K
Age
8mo
Commits · 90 Days
#7081 · +0 · 1d ·GitHub

Many thanks to Protobi and Michael for this contribution! Merged fix: styles rendering in case when "numFmt" is present in conditional formatting rules (resolves #1814) #1815.

Creator
exceljs
Size
21.3 MB
Stars
15.5K
Age
11y
Commits · 90 Days
#7122 · +0 · 1d ·GitHub

Logstash is part of the Elastic Stack along with Beats, Elasticsearch and Kibana. ). Logstash has over 200 plugins, and you can write your own very easily as well.

Creator
elastic
Size
132.2 MB
Stars
15K
Age
15y
Commits · 90 Days
#7265 · +0 · 1d ·GitHub

Papa Parse is the fastest in-browser CSV (or delimited text) parser for JavaScript. It is reliable and correct according to RFC 4180, and it comes with these features:

Creator
mholt
Size
14.5 MB
Stars
13.6K
Age
13y
Commits · 90 Days
#7713 · +0 · 1d ·GitHub
archived

xsv is a command line program for indexing, slicing, analyzing, splitting and joining CSV files. Commands should be simple, fast and composable:

Creator
BurntSushi
Size
0.5 MB
Stars
10.8K
Age
12y
Commits · 90 Days
#8625 · +0 · 1d ·GitHub

SolidityScanLegend enables real-time, high-performance data extraction and analysis through scalable, auto-scaling architecture and efficient query processing.

Size
<0.1 MB
Stars
118
Updated
CommitSep 01
Age
1mo
Commits · 90 Days
#8630 · +0 · 1d ·GitHub

NFTAPIInfinite is aimed at developers who need a straightforward, dependable solution.

Creator
lgomesappj
Size
<0.1 MB
Stars
117
Updated
CommitSep 01
Age
1mo
Commits · 90 Days
#8639 · +0 · 1d ·GitHub

MetaBridgeDiamond enables high-performance real-time data processing and auto-scaling, deploying scalable agents as a robust manager.

Creator
minseochoj
Size
<0.1 MB
Stars
114
Updated
CommitSep 01
Age
1mo
Commits · 90 Days
#8658 · +0 · 1d ·GitHub

把听到的中文名字对到一份花名册上:按音节比拼音、多音姓、口音归一、字形兜底;拿不准时给候选和理由,不替你下结论。零依赖。

Size
0.1 MB
Stars
104
Updated
CommitOct 16
Age
22d
Commits · 90 Days
#8668 · +0 · 1d ·GitHub

Enables real-time data processing and auto-scaling for cloud-based systems and modules. kept simple: just Cloud agnostic deployment

Creator
vinamega
Size
<0.1 MB
Stars
83
Updated
CommitAug 28
Age
1mo
Commits · 90 Days