LiPD Format Reference

Overview

A LiPD file (.lpd) is a ZIP archive that bundles tabular proxy data with rich structured metadata, following the BagIt specification for data integrity. The metadata lives in a JSON-LD file (metadata.jsonld); data values live in one or more headerless CSV files.

LiPD is supported by lipdR (R), pylipd (Python), and the LiPD Playground (browser).

File Structure

Inside every .lpd ZIP archive:

myfile.lpd  (ZIP archive)
└── bag/
    ├── bagit.txt              ← BagIt version declaration
    ├── bag-info.txt           ← bagging date, software agent
    ├── manifest-md5.txt       ← MD5 checksums of data/ files
    ├── tagmanifest-md5.txt    ← MD5 checksums of tag files
    └── data/
        ├── metadata.jsonld    ← all metadata (JSON-LD)
        ├── *.csv              ← one CSV per table
        └── ...

BagIt files

MD5 checksums must be recomputed whenever metadata.jsonld or any CSV is modified. The manifest format is one line per file: <md5hex> data/filename.csv

Metadata Fields

All metadata lives in data/metadata.jsonld as a JSON-LD object.

Required fields

Recommended fields

Geo Object

The geo object uses GeoJSON Feature syntax:

{
  "type": "Feature",
  "geometry": {
    "type": "Point",
    "coordinates": [longitude, latitude, elevation]
  },
  "properties": {
    "siteName": "Palmer Deep",
    "country": "Antarctica",
    "ocean": "Southern Ocean",
    "continent": "Antarctica",
    "countryOcean": "Southern Ocean",
    "locationType": "marine",
    "description": "Continental shelf basin",
    "notes": "Core taken at 1011 m water depth"
  }
}

• coordinates: [longitude, latitude, elevation] — elevation is optional; negative = below sea level
• latitude: −90 to 90
• longitude: −180 to 180
• lipdR stores geo.latitude and geo.longitude in memory but writes the GeoJSON form to disk
• locationType values: "marine", "lacustrine", "terrestrial", "ice", etc.

Publications

pub is an unnamed array at the root level. Each entry:

{
  "author": [{"name": "Smith, J."}, {"name": "Jones, A."}],
  "title": "My paleotemperature record",
  "year": 2024,
  "journal": "Paleoceanography",
  "doi": "10.1029/2024PA000001",
  "volume": "39",
  "issue": "2",
  "pages": "1-22",
  "abstract": "We present...",
  "citeKey": "smith2024paleo",
  "publicationType": "article-journal",
  "urls": ["https://doi.org/10.1029/2024PA000001"],
  "dataUrls": ["https://pangaea.de/10.1594/PANGAEA.12345"]
}

PaleoData / ChronData

Both use the same nested structure. paleoData holds proxy measurements; chronData holds age and chronological data.

"paleoData": [
  {
    "measurementTable": [
      {
        "tableName": "paleo0measurement0",
        "filename": "MyDataset.paleo0measurement0.csv",
        "missingValue": "nan",
        "columns": [ ... ]
      }
    ],
    "model": [
      {
        "summaryTable":      [ ... ],
        "ensembleTable":     [ ... ],
        "distributionTable": [ ... ]
      }
    ]
  }
]

• paleoData and chronData are unnamed arrays, not keyed objects
• measurementTable, ensembleTable, summaryTable, distributionTable are also arrays
• Table name convention (zero-indexed): paleo0measurement0, chron0model0ensemble0
• filename references the CSV in the same data/ directory
• missingValue defaults to "nan" if absent
• values is not stored in the JSON file — data lives in the CSV

Column Object

Each item in a columns array:

{
  "number": 3,
  "variableName": "SST",
  "TSid": "LPD1028d531",
  "units": "degC",
  "description": "Sea surface temperature",
  "proxy": "TEX86",
  "proxyGeneral": "biomarker",
  "inferredVariableType": "temperature",
  "hasMinValue": 15.3,
  "hasMaxValue": 28.5,
  "hasMeanValue": 22.1,
  "hasMedianValue": 22.0,
  "hasResolution": {
    "hasMinValue": 20.0,
    "hasMeanValue": 100.0,
    "hasMedianValue": 80.0,
    "hasMaxValue": 500.0
  },
  "uncertainty": {"analytical": 0.5},
  "calibration": { ... },
  "physicalSample": { ... },
  "interpretation": [
    {
      "variable": "temperature",
      "variableDetail": "surface",
      "seasonality": "Annual",
      "direction": "positive",
      "scope": "climate"
    }
  ]
}

Calibration Object

Documents the transfer function used to convert proxy values to climate variables:

{
  "equation": "SST = (TEX86 - 0.10) / 0.015",
  "equationSlope": 0.015,
  "equationIntercept": -0.10,
  "equationSlopeUncertainty": 0.001,
  "equationR2": 0.87,
  "method": "linear regression",
  "methodDetail": "Calibrated against modern core-top assemblages",
  "uncertainty": 1.5,
  "proxyDataset": "TEX86",
  "targetDataset": "SST",
  "doi": "10.1016/j.gca.2002.07.003",
  "seasonality": "Annual",
  "notes": "Kim et al. (2010) calibration"
}

• equation: full equation as a string
• equationSlope / equationIntercept: numeric regression coefficients, more machine-readable than equation alone
• equationR2: goodness of fit
• uncertainty: calibration uncertainty in units of the target variable
• doi: DOI of the calibration publication

Physical Sample Object

Links a column measurement to its physical sample, enabling sample-level provenance:

{
  "hasname": "DSDP-594-1H-1",
  "hasidentifier": "IGSN:AU1234567",
  "housedat": "IODP Bremen Core Repository"
}

CSV Files

Each measurement or model table has a corresponding .csv file in data/.

• No header row — column positions are set by each column's number field (1-indexed)
• Missing values are written as NaN (case-insensitive on read)
• String columns (e.g. labID, materialDated) may be quoted or unquoted
• All columns in a table must have the same row count
• Filename convention: DatasetName.paleo0measurement0.csv

Example — depth (col 1), age (col 2), SST (col 3):

0.11,80.0,18.5
0.21,134.0,19.2
0.31,189.0,NaN
0.41,243.0,17.8

Ensemble Table Format

Ensemble tables store large numbers of age model ensemble members (often 1000+). Rather than defining one column object per member, a single column uses an array for its number field, encoding all member column positions at once:

{
  "tableName": "chron0model0ensemble0",
  "filename": "MyDataset.chron0model0ensemble0.csv",
  "columns": [
    {
      "number": 1,
      "variableName": "depth",
      "TSid": "LPDabc12345",
      "units": "cm"
    },
    {
      "number": [2, 3, 4, 5, ..., 1001],
      "variableName": "age",
      "TSid": "LPDdef67890",
      "units": "yr BP"
    }
  ]
}

The CSV for this table has 1001 columns: column 1 is depth, columns 2–1001 are the 1000 age ensemble members.

• On read: expand the array-number column into individual column objects (one per index); preserve the original array for exact round-trip reconstruction
• On write: collapse individual expanded columns back into a single column object with number as the array
• This pattern is used by lipdR and pylipd; any parser must handle it

Changelog Entry Formats

Two styles appear in real files. Both must be handled on read.

lipdR style (structured changes)

{
  "version": "1.1.0",
  "date": "2024-03-15",
  "name": "Researcher Name",
  "notes": "Added TEX86 calibration",
  "changes": {
    "addedColumn": [["SST added to paleo0measurement0"]],
    "editedMetadata": [["Updated archiveType"], ["Fixed DOI"]]
  }
}

The changes field is an object whose keys are change-type labels. Values are arrays of arrays of strings (matrix format used by lipdR).

pylipd / manual style (flat notes)

{
  "version": "1.1.0",
  "timestamp": "2024-03-15T10:30:00Z",
  "curator": "Researcher Name",
  "notes": "Added TEX86 calibration and fixed DOI",
  "lastVersion": "1.0.0"
}

Here timestamp supplements or replaces date; curator supplements or replaces name; lastVersion records the previous version.

LiPD Versions

lipdR and pylipd auto-upgrade older versions to 1.3 on read.

Time Series IDs (TSid)

Every column must have a unique TSid. It serves as a global identifier for the time series across the LiPD ecosystem. TSids must be unique within a dataset and ideally globally unique. No specific format is enforced — any unique string is valid.

JSON-LD Context and Ontology

LiPD files use JSON-LD. The @context field in metadata.jsonld typically references "context.jsonld" (a relative path inside the ZIP).

The primary semantic framework is the LinkedEarth ontology:

• Namespace: http://linked.earth/ontology#
• Current version: 1.2.0