Importing Data

The data block defines where data is to be loaded form, and how it will be loaded and transformed.

The specification of data loading in NetPanorama is very similar to that in Vega, and NetPanorama reuses the vega-loader library (GitHub, npm). Like Vega (and unlike Vega-lite), the data block contains a list of named data sources, rather than a single data source.

Data can be loaded from JSON or character-delimited files (CSV, TSC, DSV). It can then be processed by applying a transformation.

In some cases, a visualization may rely on the several different transformations of the same dataset. To reduce repetition (and prevent the data from being fetched multiple times), you can instead provide the name of an existing dataset as the source of a dataset.

caution

Whilst the API is very similar to Vega, there are differences in what transformations are supported. Additionally, loading data from LocalStorage is a feature specific to NetPanorama.

Limitation

It is not currently possible to load data dynamically at runtime through a view API.

Examples

Data defined inline:

{ "name": "table", "values": [12, 23, 47, 6, 52, 19] }

Data loaded from a URL:

{ "name": "points", "url": "data/points.json" }

Loading a dataset that was saved to LocalStorage using the Upload Data button in the editor:

{ "name": "networkData", "localStorage": "networkData" }

Importing a CSV file, and parsing one column as a Date:

{
    "name": "marieboucher",
    "url": "./data/marieboucher.csv",
    "format": {"type": "csv", "parse": {"Date": "date:'%d/%m/%Y'"}}
}

Creating a new derived dataset from an existing dataset:

{
  "name": "stats",
  "source": "table",
  "transform": [
    {
      "type": "aggregate",
      "groupby": ["x"],
      "ops": ["average", "sum", "min", "max"],
      "fields": ["y", "y", "y", "y"]
    }
  ]
}

Transformations

When defining a data source, the transformations attribute can be used to define a list of transformations that should be applied to the data.

It is also possible to make a transformed copy of dataset by defining a new dataset, with source set to the name of the existing dataset. For example,

    {
  "name": "roles",
  "source": "credits",
  "transform": [
    {
      "type": "unroll",
      "fields": [
        "cast",
        "crew"
      ]
    }
  ]
}

caution

Transformations that require understanding of the network structure of a dataset (i.e., that an edge connects a source and target nodes), rather than treating the data as simply a table, are instead defined in the network block.

Adding columns or fields

Add Index

Add a sequential id field to each element in a data array.

Attribute	Type	Description
type	"addIndex "
as?	string	The name of the field in which to record the id.

Lookup

Extend the elements in a (primary) data array by looking up additional values in another (secondary) data array.

Attribute	Type	Description
type	"lookup"
from	string	The name of the data array to join with.
key	string	The field in the secondary array containing the key that will be used for matching.
fields	string[]	The fields in the primary data array that will be matched against the key in the secondary array.
values	string[]	The fields on the secondary data array hose values should be copied.
as?	string[]	The name of the fields on the primary data array in which to store the looked-up values.
default	any	The default value to use if lookup fails.

Geocode

Geocode locations, by finding a latitude and longitude from a place name.

Attribute	Type	Description
type	"geocode"
locationField?	string	The name of a field containing a location name.
streetField?	string	The name of a field containing a street name.
cityField?	string	The name of a field containing a city name.
stateField?	string	The name of a field containing a state name.
countryField?	string	The name of a field containing a country name.
postalCodeField?	string	The name of a field containing a postal code.
lonAs?	string	The name of a field in which the longitude should be saved.
latAs?	string	The name of a field in which the latitude should be saved.
nameAs?	string	If specified, the displayName of the geocoded location will be recorded as a field of this name.This can be helpful in identifying where the geocoder has picked the wrong location for an ambiguous place name.

Geocoding uses the API provided by https://geocode.maps.co, which is based on OpenStreetMap and Nominatim. The results of queries are saved in the browser's LocalStorage, so that identical queries are not made repeatedly.

Calculate

Evaluate an expression, and record the resulting value as a field.

Attribute	Type	Description
type	"calculate"
calculate	string	The expression to evaluate.
as	string	The name of the field in which to record the calculated value.

Transforming values

Bin values

Group continuous numerical data into discrete bins.

Attribute	Type	Description
type	"bin"
field	string	The field to be binned.

Similar to Vega-Lite's Bin transform

Removing rows

Filter

Filter a list of data items, keeping only those for which the specified expression evaluates to true.

Attribute	Type	Description
type	"filter"
expr	string	The expression to evaluate for each data item.

Sample

Select a random subset of a data array.

Attribute	Type	Description
type	"sample"
sample	number	The number of data elements to keep.

N.B., a similar thing could be achieved by using a filter transformation with an expression like random() < 0.5, but the exact number of items that this would produce is not guaranteed.

Reshaping

Aggregate

Aggregate rows of the data table that have the same values in one (or multiple) columns.

Attribute	Type	Description
type	"aggregation"
groupby?	string[]	The data fields to group by.
fields?	string[]	The data fields for which to compute aggregate functions.
ops?	AggregationFunc[],	The aggregation functions to apply.
as?	string,	The field names to use for the computed aggregate fields.

Similar to Vega's Aggregate transform.

Flatten

Maps array-valued fields to a set of individual data objects, one per array entry.

Attribute	Type	Description
type	"flatten"
flatten	string[]	Array containing the names of fields to flattten.

For example, flattening on the fields ["foo", "bar"] will transform the input data:

[
  {
    "key": "alpha",
    "foo": [
      1,
      2
    ],
    "bar": [
      "A",
      "B"
    ]
  },
  {
    "key": "beta",
    "foo": [
      3,
      4,
      5
    ],
    "bar": [
      "C",
      "D"
    ]
  }
]

into:

[
  {
    "key": "alpha",
    "foo": 1,
    "bar": "A"
  },
  {
    "key": "alpha",
    "foo": 2,
    "bar": "B"
  },
  {
    "key": "beta",
    "foo": 3,
    "bar": "C"
  },
  {
    "key": "beta",
    "foo": 4,
    "bar": "D"
  },
  {
    "key": "beta",
    "foo": 5,
    "bar": undefined
  }
]

Fold

Collapses one or more data fields into a key and value.

Attribute	Type	Description
type	"fold"
fields	string[]	Array containing the name of fields to fold.

For example, flattening on the fields ["centrality", "degree"] will transform the input:

[
  {
    id: 0,
    centrality: 2,
    degree: 1
  },
  {
    id: 1,
    centrality: 5,
    degree: 6
  }
]

into:

[
  {
    field: 'centrality',
    value: 2,
    id: 0,
    centrality: 2,
    degree: 1
  },
  {
    field: 'centrality',
    value: 5,
    id: 1,
    centrality: 5,
    degree: 6
  },
  {
    field: 'degree',
    value: 1,
    id: 0,
    centrality: 2,
    degree: 1
  },
  {
    field: 'degree',
    value: 6,
    id: 1,
    centrality: 5,
    degree: 6
  }
]

Pivot

Maps unique values from a field to new aggregated fields.

Attribute	Type	Description
type	"pivot"
pivot	string	The name of the field to pivot on; the unique values of this field become new field names in the output.
value	string	The name of the field to populate pivoted fields; the aggregate values of this field become the values of the new pivoted fields.
groupby	string[]	Names of (optional) fields to group by.
op	string	The aggregation operation to apply.

For example, the transformation:

{
  "type": "pivot",
  "pivot": "type",
  "groupby": [
    "country"
  ],
  "value": "count"
}

will transform the input data:

[
  {
    "country": "Norway",
    "type": "gold",
    "count": 14
  },
  {
    "country": "Norway",
    "type": "silver",
    "count": 14
  },
  {
    "country": "Norway",
    "type": "bronze",
    "count": 11
  },
  {
    "country": "Germany",
    "type": "gold",
    "count": 14
  },
  {
    "country": "Germany",
    "type": "silver",
    "count": 10
  },
  {
    "country": "Germany",
    "type": "bronze",
    "count": 7
  },
  {
    "country": "Canada",
    "type": "gold",
    "count": 11
  },
  {
    "country": "Canada",
    "type": "silver",
    "count": 8
  },
  {
    "country": "Canada",
    "type": "bronze",
    "count": 10
  }
]

into:

[
  {
    "country": "Norway",
    "gold": 14,
    "silver": 14,
    "bronze": 11
  },
  {
    "country": "Germany",
    "gold": 14,
    "silver": 10,
    "bronze": 7
  },
  {
    "country": "Canada",
    "gold": 11,
    "silver": 8,
    "bronze": 10
  }
]

(this example is borrowed from the Vega-Lite documentation).

Examples​

Transformations

Adding columns or fields​

Add Index​

Lookup​

Geocode​

Calculate​

Transforming values​

Bin values​

Removing rows​

Filter​

Sample​

Reshaping​

Aggregate​

Flatten​

Fold​

Pivot​

Examples

Adding columns or fields

Add Index

Lookup

Geocode

Calculate

Transforming values

Bin values

Removing rows

Filter

Sample

Reshaping

Aggregate

Flatten

Fold

Pivot