Many applications use JSON as their output format for structured
information. I think there should be tool that allows processing JSON
using Relational Pipes or outputting processed data to JSON.
There is also similar format/language called YAML ( https://yaml.org )
that is extension of JSON. Its syntax allows writing structured data in
“block syntax” that is easily readable even by inexperienced users. This
format is becoming favorite configuration and log file format for many
Since every JSON file is valid YAML we could create one program
(relpipe-in-yaml) that converts YAML/JSON input containing tabular data
(similar to the second example) into Relational Pipes stream.
Users that don't know YAML will probably look for relpipe-in-json. this
program could be symlinked to relpipe-in-yaml.
JSON and block-style YAML output is relatively simple, and it can be
done without external libraries.
Examples of YAML
These examples are copied from YAML 1.2 specification.
( https://yaml.org/spec/1.2/spec.html )
╭── Example 2.27. Invoice
│ --- !<tag:clarkevans.com,2002:invoice>
│ invoice: 34843
│ date : 2001-01-23
│ bill-to: &id001
│ given : Chris
│ family : Dumars
│ lines: |
│ 458 Walkman Dr.
│ Suite #292
│ city : Royal Oak
│ state : MI
│ postal : 48046
│ ship-to: *id001
│ - sku : BL394D
│ quantity : 4
│ description : Basketball
│ price : 450.00
│ - sku : BL4438H
│ quantity : 1
│ description : Super Hoop
│ price : 2392.00
│ tax : 251.42
│ total: 4443.52
│ Late afternoon is best.
│ Backup contact is Nancy
│ Billsmer @ 338-4338.
╭── Example 2.28. Log File
│ Time: 2001-11-23 15:01:42 -5
│ User: ed
│ This is an error message
│ for the log file
│ Time: 2001-11-23 15:02:31 -5
│ User: ed
│ A slightly different error
│ Date: 2001-11-23 15:03:17 -5
│ User: ed
│ Unknown variable "bar"
│ - file: TopClass.py
│ line: 23
│ code: |
│ x = MoreObject("345\n")
│ - file: MoreClass.py
│ line: 58
│ code: |-
│ foo = bar
Let us start some brainstorming!
We have both input and output filters for CSV. This is a quite nice format, concise and human readable, but has some limitations:
1) only one relation in the file/stream 2) data types are lost - the CSV contains only the attribute name, not the type
The 1) is inherent insoluble problem. But what about 2) ? Currently we can specify the types on CLI:
cat file.csv | relpipe-in-csv --attribute first integer --attribute second string
but it is not comfortable, we have to skip the header if present (e.g. add tail -n +2 to the pipeline) and the CLI arguments must match the structure of the file.
Sometimes it might be useful to store the type information in the CSV and then be able to automatically parse the types while reading. How?
a) fixed format like "first:integer","second:string" i.e. "attribute_name:data_type"
b) let the user specify the regular expression to extract the attribute name and type – there could be any separator and any order of the name/type
c) prefix notation – some software e.g. recognize "+abc" as a string attribute, "-abc" as numeric one, "d+abc" as date
c) use regular expressions to describe name patterns e.g. say ".*_count|size|age" attributes are integers and ".*_flag|enabled|deleted" are booleans
d) store attribute names in the first line and data types in the second
Any other ideas?
I started writing packages for the Gentoo Linux distribution.
Gentoo is distribution that uses only source-code packages (well, there
is support for binary packages, but typical Gentoo packages are
compiled). The package is described by its ebuild file that contains
metadata and instructions to build it. Portage, package manager of
Gentoo, uses USE flags to provide additional configuration of packages.
Ebuilds for Relational Pipes are divided into two sections: app-misc
(in, out, and tr modules) and dev-libs (libraries). If you can find
category that fits them better, let me know.
Complete list of available categories:
Naming scheme of packages for Gentoo:
· dev-libs/relpipe-* for libraries
app-misc/relpipe-tr-* for executables
app-misc/relational-pipes-tr are empty packages that depend on
· app-misc/relational-pipes depends on all modules of Relational Pipes
app-misc/relational-pipes-* also have USE flags that modify their
dependencies. Table of USE flags for app-misc/relational-pipes-in:
USE flag Added dependency
These use flags allow the user install only part of the Relational Pipes
framework without need to install the packages separately.
What license do you prefer for the package files? LGPL 3 is probably
I am creating packages for Gentoo, and compilation of file
`src/RelpipeChartMainWindow.cpp` in `relpipe-out-gui.qt.cpp` was failing
with the following error:
invalid use of incomplete type ‘class QHeaderView’
I suggest adding this include directive to file
We are pleased to introduce you the new development version of Relational pipes. This release improves and standardizes the CLI interface of several tools, brings a new output module + several smaller improvements.
* The relpipe-in-cli module has changed CLI interface
* The relpipe-in-csv module has changed CLI interface
* The relpipe-out-csv module has new CLI interface
* The relpipe-out-tabular module has new CLI interface
* The relpipe-tr-guile was renamed to relpipe-tr-scheme
* New tool relpipe-out-jack for generating MIDI through JACK
See full release notes <https://relational-pipes.globalcode.info/v_0/release-v0.17.xhtml>, examples <https://relational-pipes.globalcode.info/v_0/examples.xhtml> and screenshots <https://relational-pipes.globalcode.info/v_0/screenshots.xhtml> pages for details.
Please note that this is still a development release and thus the API (libraries, CLI arguments, formats) might and will change. Any suggestions, ideas and bug reports are welcome in this mailing list.