Skip to content

Parquet input

Plot data accepts Parquet alongside CSV, pandas text output and other formatted tables.

AI-assisted, human-approvednovem uses AI to review and keep our documentation up to date.

Send a Parquet file to an existing plot's data endpoint, using the same URL and write permissions as a text upload:

curl --fail-with-body \
  -H "Authorization: Bearer $NOVEM_TOKEN" \
  -H "Content-Type: application/vnd.apache.parquet" \
  --data-binary @data.parquet \
  https://api.novem.io/v1/vis/plots/myplot/data

The API also accepts application/x-parquet and application/octet-stream for Parquet. Send CSV, pandas text output and other formatted text tables with Content-Type: text/plain. The text parser keeps its existing delimiter, header and value detection.

The canonical URL is /v1/users/:user/vis/plots/:plot/data. The shorthand /v1/vis/plots/:plot/data addresses your own account. Both require the plot's normal write authority, token scopes, route grants and subscription eligibility. If-Match works as for other plot data writes. A write-only recipient can upload data without gaining access to the plot's contents.

Parsed values

A Parquet upload provides the complete table in source row and column order. The schema supplies the header and column types. Charts use column names as headers, and tables keep the header as row zero, following the existing cell selector layout.

Read the parsed values at .../json/data.json and column metadata at .../json/meta_data.json. Parquet metadata includes source_columns. Reading .../data returns the original Parquet bytes with Content-Type: application/vnd.apache.parquet. Text uploads return their original text at the same URL.

Parquet typePlot values
BooleanJSON true or false
Signed and unsigned integersJSON numbers within ±9007199254740991. Exact decimal strings outside that range
FLOAT, DOUBLEFinite JSON numbers. NaN and infinities are rejected
DECIMALExact fixed-point strings, retaining scale, up to 38 digits of precision and scale
STRING, UTF8, ENUMUTF-8 strings, including whitespace, newlines and numeric-looking text
DATEISO calendar dates
TIMESTAMPISO date and time with source millisecond, microsecond or nanosecond precision
Unannotated binary and fixed-length binaryBase64 strings
NullJSON null, distinct from an empty string and the literal string "null"

UTC-adjusted timestamps include Z. Local timestamps retain their wall-clock value without a timezone suffix. Timestamp values remain text for axis classification. Dates are date columns. Exact numeric strings survive in the input and parsed data. Chart coordinates and numeric styling use floating-point arithmetic.

Supported files and limits

Single-file flat tables are decoded by visparse using PyArrow. Plain, dictionary, delta and byte-stream-split encodings are supported with uncompressed, Snappy, Gzip, Brotli, Zstd or LZ4 pages. Nested lists, maps and structs, external column files, legacy INT96 timestamps, time-only values and encrypted files are unsupported. Flatten nested data and convert unsupported columns before uploading.

The existing plot data size limit applies to uploaded bytes, including streaming uploads without a content length. Decoding limits a table to 100,000 rows, 256 columns and 1,000,000 cells, with a 2 MiB footer and 4096 row groups. Declared uncompressed data and decoded Arrow values are each capped at 64 MiB.

Oversized uploads return 413 before changing the stored input. As with text input, the source is saved before visparse processes it. Corrupt files, unsupported types and decoded-data limits return a 400 parse error. The uploaded source remains available at .../data, and the previous parsed output remains until a valid input is processed. A write-only caller gets no parser diagnostics or read access.