datatable is a Python package for manipulating 2-dimensional tabular data structures (aka data frames). It supports out-of-memory datasets, multi-threaded data processing, and flexible API.
It is close in spirit to pandas or SFrame; however we put specific emphasis on speed and big data support. As the name suggests, the package is closely related to R’s data.table and attempts to mimic its core algorithms and API.
This is free and open source software.
Goals:
- Column-oriented data storage.
- Native-C implementation for all datatypes, including strings. Packages such as pandas and numpy already do that for numeric columns, but not for strings.
- Support for date-time and categorical types. Object type is also supported, but promotion into object discouraged.
- All types should support null values, with as little overhead as possible.
- Data should be stored on disk in the same format as in memory. This will allow us to memory-map data on disk and work on out-of-memory datasets transparently.
- Work with memory-mapped datasets to avoid loading into memory more data than necessary for each particular operation.
- Fast data reading from CSV and other formats.
- Multi-threaded data processing: time-consuming operations should attempt to utilize all cores for maximum efficiency.
- Efficient algorithms for sorting/grouping/joining.
- Expressive query syntax (similar to data.table).
- Minimal amount of data copying, copy-on-write semantics for shared data.
- Use “rowindex” views in filtering/sorting/grouping/joining operators to avoid unnecessary data copying.
- Interoperability with pandas / numpy / pyarrow / pure python: the users should have the ability to convert to another data-processing framework with ease.
Website: github.com/h2oai/datatable
Support:
Developer: datatable developers
License: Mozilla Public License 2.0
datatable is written in C++ and Python. Learn C++ with our recommended free books and free tutorials. Learn Python with our recommended free books and free tutorials.
Related Software
| Python Data Analysis | |
|---|---|
| pandas | High-level building block for doing practical, real world data analysis |
| NumPy | Core package for scientific computing with Python |
| SciPy | Ecosystem for mathematics, science, and engineering |
| Polars | DataFrame interface on top of an OLAP Query Engine |
| statsmodels | Statistical modeling and econometrics in Python |
| Dask | Advanced parallelism for analytics |
| Orange | Component-based framework for machine learning and data mining |
| Modin | Drop-in replacement for pandas |
| Vaex | Fast visualization of big data |
| AWS DW | Extends the power of pandas library |
| yt | Multi-code Toolkit for Analyzing and Visualizing Volumetric Data |
| HoloViews | Make Data Analysis and Visualization Seamless |
| datatable | Manipulate 2-dimensional tabular data structures |
| xarray | Work with labelled multi-dimensional arrays and datasets |
| pyjanitor | Extend pandas with readable data-cleaning functions |
| Optimus | Agile Data Preparation Workflows |
Read our verdict in the software roundup.
 Explore our comprehensive directory of recommended free and open source software. Our carefully curated collection spans every major software category.This directory is part of our ongoing series of informative articles for Linux enthusiasts. It features hundreds of detailed reviews, along with open source alternatives to proprietary solutions from major corporations such as Google, Microsoft, Apple, Adobe, IBM, Cisco, Oracle, and Autodesk. You’ll also find interesting projects to try, hardware coverage, free programming books and tutorials, and much more. Discovered a useful open source Linux program that we haven’t covered yet? Let us know by completing this form. |