Geodata Processing
This page provides on overview of the processing paths that may be applied to geodata, using the Data Processing API. This includes processing of 3rd-party 2D image pixel data, 2D vector shape data, 3D model data as well as Tinman-specific raster data and pyramid data.
Overview
When working with geodata, the initial step is to determine which geodata interface to use, for example Pixel Data or Shape Data.
The next step is to obtain an object that implements the interface, while wrapping the geodata. For example, an IImage or IShape object.
The final step is to combine these geodata objects into a processing pipeline, for example to produce Raster Data and Pyramid Data.
The next sections each contain the following sub-sections, which describe each of the above steps:
- Import
-
Shows how to wrap an existing data source with the API, in order to obtain an instance of a geodata interface. For example:
"Open geo-referenced image file <my-data.tif>." - Consume
-
Shows how to use the API to modify and combine geodata objects and how to access the resulting geodata. For example:
"Read pixel data after removing border pixels." - Process
-
Shows ways to assemble geodata objects into a processing pipeline and to otherwise use them in the API. For example:
"Transfer pixel data into a raster or pyramid dataset."
"Create a cubemap texture from a DDS file."
The above classification is ambiguous in some cases, for example: importing 3D model data by extruding 2D shape data could also be seen as processing 2D shape data into 3D model data by extruding it. In these cases, you will usually find the description in the Process step. |
The following table shows the geodata interfaces and corresponding links to the documentation on how to obtain geodata objects and how to process them.
Geodata | Interface | Documentation |
---|---|---|
Pixel data |
See Pixel Images domain model and Pixel Data section. |
|
Shape data |
See 2D Shapes domain model and Shape Data section. |
|
Model data |
See 3D Models domain model and Model Data section. |
|
Raster data |
See Raster overview and Raster Data section. |
|
Pyramid data |
See Pyramid overview and Pyramid Data section. |
Since geodata is accessed through interfaces, you may implement your own implementations to wrap geodata sources that are not supported by the Tinman 3D SDK. |
While combining geodata objects, it is often the case that transformations and conversions need to be performed between geodata objects, possibly involving different geodata interfaces. The following table gives an overview of the most common conversion paths.
From / To | IImage | IShape | IModel | IHeightmap | IPixelPyramid |
---|---|---|---|---|---|
(1) |
(2) |
||||
(3) |
(4) |
(3) |
(3) |
(3) |
|
(5) |
(6) |
||||
(7) |
(7) |
(8) |
(7) |
||
(9) |
(10) |
1 | See Pixel Data / Consume section. |
2 | See Pixel Data / Process section. |
3 | See Shape Data / Process section. |
4 | See Shape Data / Consume section. |
5 | See Model Data / Consume section. |
6 | See Model Data / Process section. |
7 | See Raster Data / Process section. |
8 | See Raster Data / Consume section. |
9 | See Pyramid Data / Process section. |
10 | See Pyramid Data / Consume section. |
Pipeline Tasks
This section describes the individual tasks of the geodata processing pipeline while referring to the involved APIs. Implementing a geodata processing pipeline for an application or a processing script usually boils down to choosing some processing tasks and to hook them up appropriately.
The GeodataPipeline class is an example for a geodata processing pipeline. |
- Task List
-
§1 - Import one pixel data source to a new raster dataset
§2 - Import pixel data sources into a raster dataset
§3 - Project data from rectangular to cubemap datasets
§4 - Convert raster data to pyramid data
§5 - Copy raster and pyramid data
§6 - Update raster and pyramid data at runtime
§7 - Replace raster and pyramid data at runtime
§8 - Combine raster and pyramid data at runtime
File Data
Geodata files are commonly referenced with Path objects and/or plain string
values.
In addition, the IFileData interface may be used to provide the binary content of a geodata file in a more flexible way.
API | Scripting | Description |
---|---|---|
Uses the given binary content. |
||
Uses the content of the given file. |
||
Uses the content of an entry in a ZIP archive. |
||
- |
Reads the file data as a binary data stream. |
Procedural Data
Procedural data can be generated with the Tinman 3D SDK by using so called noise functions. A noise function takes a three-dimensional input vector and produces a smooth pseudo-random output value for it, i.e. the output value is deterministic and depends only on the input vector.
There are three types of noise functions:
-
IColorFunction (produces color values)
-
IScalarFunction (produces scalar values)
-
IVectorFunction (produces vector values)
All noise functions are derived from the INoiseFunction interface, which provides common semantics:
-
Noise function objects are typically arranged in an object graph, which involves arithmetic operations and other transformations.
-
To minimize the complexity of such object graphs, noise function objects provide a way to simplify the object graph structure, usually according to mathematical rules like
A + 0 = A
orA * 0 = 0
. -
Constant noise functions always produce the same output value regardless of the input vector. This information is crucial to simplification.
All noise function objects use the low-level noise functions of the ProceduralUtil class. |
The following table shows how to obtain noise function objects.
API | Scripting | Description |
---|---|---|
Outputs the given constant color value. |
||
Outputs Cellular noise scalar values. |
||
Outputs the given constant scalar value. |
||
Outputs the constant scalar value |
||
Outputs Perlin noise scalar values. |
||
Outputs the given constant vector value. |
||
Outputs gradient vector values of Perlin noise scalar values. |
||
Outputs the result of a linear transformation of the input vector value. |
The following table shows how to combine noise function objects.
API | Scripting | Description |
---|---|---|
IColorFunction.Add |
Applies the given operator on the input value(s). |
|
Scalar.Constant |
Applies a linear function on each component of the output value. |
|
- |
Simplifies the underlying object graph structure of the noise object. |
|
Uses a vector function to transform the input value. |
||
IScalarFunction.Abs |
Applies the given operator on the input value(s). |
|
Clamps the output value to the given range. |
||
Uses the output value of a scalar function to perform a color-ramp lookup, in order to produce the output color value. |
||
- |
Applies the given operator on the input value(s), swapping the operands. |
|
Computes noise over multiple octaves. |
||
Computes noise over multiple octaves. |
||
Replaces not-a-number and +/- infinity with the given values. |
||
- |
Applies the given operator on the input value(s), swapping the operands. |
|
Uses the output value of a scalar function to perform cubic interpolation on an array of aggregated scalar functions. |
||
Uses the output value of a scalar function to evaluate a spline curve. |
||
- |
Applies the given operator on the input value(s), swapping the operands. |
|
Uses a scalar function to compute an offset for each component of the input value. |
||
IVectorFunction.Add |
Applies the given operator on the input value(s). |
|
Computes noise over multiple octaves. |
Pixel Data
Pixel data in the Tinman 3D SDK is defined as a rectangle that is made of square pixels, where each pixel may either be void (i.e. not representing any data) or represent a color value with separate red, green, blue and alpha (transparency) channels, each having 24-bits of integer precision. These are called normalized pixel values and range from 0 to 16777215 (i.e. 224-1).
Pixel data sources (for example image files) use varying data types and value ranges for their raw pixel data, for example 8-bit unsigned integers for sRGB color channel values or 32-bit floating-point values for elevation values in metres. The mapping from the raw pixel range to the normalized pixel range is retained in IImageInfo.Range, so that it may be taken into account in subsequent processing steps.
For additional information, please refer to Software Architecture / Pixel Images.
Import
Importing pixel data means to wrap a pixel data source in an IImage object, in order to be able to access the image metadata and image content.
API | Scripting | Description |
---|---|---|
Generates an image with constant pixel values. |
||
Reads pixel data from an image file. |
||
Generates an image with random pixel data. |
||
Reads raw pixel data from binary data. |
Consume
Consuming pixel data means to read the content of an IImage object by using the ImageReader object that has been obtained via IImage.ReadImage. Pixel data is retrieved by calling IImageReader.ReadScanline repeatedly, once for each pixel row.
Various on-the-fly conversions and transformations of pixel data may be performed by wrapping an IImage or IImageReader object in another one, which the modifies the pixel values of the Scanline object. The IImageOps interface defines a set of built-in pixel data modifications.
API | Scripting | Description |
---|---|---|
Applies a sharpening filter to the pixel data. |
||
Adjusts brightness and/or saturation. |
||
Replaces transparent pixels with void pixels. |
||
Extracts a channel as a grayscale image. |
||
Extracts a sub-rectangle as an image. |
||
Replaces border pixels with void pixels. |
||
Applies a function on the pixel values of a channel. |
||
Replaces a channel with pixel values of a grayscale image. |
||
Inverts the pixel values of a channel. |
||
Combines two images using the chosen color operation. |
||
Duplicates edge pixels to increase the image size. |
||
Scales an image to a different size. |
||
Reinterprets the pixel data with the given pixel coverage. |
||
Configures gamma correction for pixel values. |
||
Reinterprets the pixel data with the given geo-reference. |
||
Specifies the heightmap layer to associate with the pixel data. |
||
Specifies the pixel value to interpret as void. |
||
Configures the raw and normalized pixel ranges. |
||
Applies a smoothing filter to the pixel data. |
||
Resamples the pixel data to the given pixel coverage. |
||
Uses separate images per channel to assemble a new image. |
||
Encodes normal vectors in the color channels of an image. |
||
Arranges multiple images in a grid to assemble a new image. |
Process
Pixel data is processed into Raster Data by transferring its pixel values into a IHeightmap object as HeightmapSample values. Various transformations may be applied during this step, usually to adjust the pixel value range and to resample / project pixels when moving from one geo-reference to another.
To export pixel data for consumption by 3rd-party software, the IImageWriter interface may be used. Please refer to the built-in ImageFormat objects for details on the supported file formats.
API | Scripting | Description |
---|---|---|
Creates a new PXB file or a HGT dataset and copies all pixel data to it, without performing resampling or reprojection. |
||
Transfers all pixel data into an existing IHeightmap object, while performing resampling and reprojection, as required by the geo-references. |
||
- |
Reads the pixel data directly as a texture resource, avoiding redundant decoding and encoding steps where possible. |
|
Exports all pixel data into an image file of the specified file format. |
||
Copies pixel data to an existing IHeightmap object, without performing resampling or reprojection. |
Another way to process pixel data is to feed it into other parts of the Tinman 3D SDK, either as IImage objects or as derived data objects, for example ColorBuffer objects (image manipulation) or TexelBuffer (GPU rendering) objects. Here is a list of possible API calls:
Shape Data
Shape data in the Tinman 3D SDK is defined as a signed distance field (SDF) in a two-dimensional coordinate system, with an optional geo-reference and optional geometry data (vertices, edges, triangles, polygon contours).
Based on this definition, shapes may be implicit (signed distance <= 0), points (only vertices), graphs (only vertices and edges), polygons (only contours made of vertices and edges) or meshes (only vertices and triangles).
For additional information, please refer to Software Architecture / 2D Shapes.
Import
Importing shape data means to wrap a shape data source in a IShape object, in order to be able to access the shape metadata, the shape geometry and the shape distance field.
API | Scripting | Description |
---|---|---|
- |
Creates shapes for pavements and ground markings. |
|
Shape.ArcInner |
- |
Creates a pre-defined shape. |
Creates a simple axis-aligned box shape. |
||
Creates a procedural checkerboard shape. |
||
Creates a procedural circle shape. |
||
Creates a shape from the specified geometry. |
||
Reads shape data from a shape file. |
Here are some other API members that provide IShape objects:
Consume
Consuming shape data means to use the signed distance field and the geometry data (vertices, edges, triangles, polygon contours), optionally taking into account the shape geo-reference.
There are various conversions and transformations that may be applied to IShape objects, for example signed distance field manipulation, projection to other geo-references and conversion between shape types (implicit, points, graph, mesh, polygon).
Many IShape implementations compute data lazily, at the time it is requested. In these cases, computation may be enforced with IShapeOps.PrepareLazy. In the following table, deferred indicates lazy computations and immediate indicates computations that are performed instantly.
API | Scripting | Description |
---|---|---|
deferred |
Modifies the signed distance field:
|
|
immediate |
- |
Converts the shape from one type to another: implicit - points - graph - polygon - mesh |
deferred |
Modifies the signed distance field:
|
|
deferred |
Modifies the signed distance field:
|
|
deferred |
Modifies the signed distance field:
|
|
deferred |
Modifies the signed distance field:
|
|
deferred |
- |
Modifies the signed distance field:
|
immediate |
- |
Converts the shape to a graph and mesh shape, for subsequent rendering with lines and triangles. |
IShapeOps.TransformCoordinateSystem deferred |
Transforms the shape geometry from one geo-reference to another. |
|
IShapeOps.TransformCoordinates deferred |
Transforms the shape geometry with an affine transformation. |
|
deferred |
Transforms the shape geometry to the faces of the geographic cubemap frame. |
|
immediate |
- |
Transforms the shape geometry from one geo-reference to another. |
Process
The most common way to process shape data is to perform on-the-fly rasterization at runtime, based on the signed distance field. This is used to generate Raster Data and Pyramid Data, for example.
API | Scripting | Description |
---|---|---|
Reads the shape data and writes it to a CSH file. |
||
- |
Constructs a IShapeRasterizer, using the specified factory. |
|
- |
Constructs 3D geometry by extruding the geometry of a 2D shape. |
|
- |
Converts a 2D shape to planar 3D geometry. |
Another way to process shape data is to feed it into other parts of the Tinman 3D SDK. Here is a list of possible API calls:
Model Data
Model data in the Tinman 3D SDK is defined as an object graph which assembles hierarchical 3D models from basic components, such as 2D textures and 3D geometry, in a way that is comparable to glTF 2.0.
For additional information, please refer to Software Architecture / 3D Models.
Import
Importing model data means to construct individual model components and then to assemble these into a hierarchical 3D model. This can be done explicitly by creating each component from scratch while specifying all required data. There are also various factories for constructing commonly used model data directly. Additionally, ready-to-use model data may be loaded from 3D model files.
API | Scripting | Description |
---|---|---|
Creates a IModelReader object will can be used to load model data, optionally providing methods for data manipulation and correction, if the object is an instance of ModelReader. |
||
- |
Loads the given model data and returns the root IModel object. |
|
- |
Builds a IModel object for an exported terrain mesh chunk. |
API | Scripting | Description |
---|---|---|
- |
Creates a ModelGeometryBuilder object that can be used to build model data programmatically. |
|
ModelGeometry.Box |
- |
Creates a IModelGeometrySimple object using a pre-defined factory method. |
- |
Creates a IModelGeometryDynamic object which can be used to hold model geometry data that may change at runtime. |
|
- |
Creates a IModelGeometrySimple object from a IShape object. |
|
- |
Wraps user-defined model geometry data in a IModelGeometry object. |
|
Creates a IModelTexture that has a uniform color. |
||
Creates a IModelTexture that encodes the pixel data of an IImage object. |
||
Creates a IModelTexture from the contents of a TexelBuffer, storing it as an in-memory TXB file. |
||
Creates a IModelTexture from the contents of a ColorBuffer, storing it as an in-memory PXB file. |
||
- |
Creates a IModelTexture that references the output texture resource of a IRenderTarget. |
Here are some other API members that provide model data objects:
Consume
Consuming model data means to access the content of the model data objects that are contained in a 3D model hierarchy.
Various on-the-fly conversions and transformations of model data may be performed by wrapping IModel, IModelGeometry, IMaterial and IModelTexture objects in other ones, which the modifies the data.
Model data objects may compute data lazily, at the time it is requested. In these cases, computation may be enforced with IModel.PrepareLazy. In the following table, deferred indicates lazy computations and immediate indicates computations that are performed instantly.
API | Scripting | Description |
---|---|---|
immediate |
- |
Attaches a GPU instance buffer to the model, thus enabling instanced rendering. |
deferred |
- |
Attaches a GPU instance buffer to the geometry, thus enabling instanced rendering. |
deferred |
- |
Removes unused vertices and reorders the remaining ones for improved rendering performance. |
deferred |
- |
Reduces floating-point precision from 64-bit to 32-bit, in order to reduce memory consumption and storage space. |
deferred |
- |
Transforms the vertices using the specified Mat4D matrix value. |
deferred |
Inverts the given color channels. |
|
deferred |
Converts a bump-map texture into a normal-map texture. |
|
deferred |
Replaces or swaps color channels, according to the given operands. |
Process
Usually, processing model data means to either render 3D models with a GPU (see ModelRenderer) or to perform spatial queries on the 3D geometry (see ISpatialQuery).
To export model data for consumption by 3rd-party software, the ModelFormat.WriteModel method may be used. Please refer to the built-in ModelFormat objects for details on the supported file formats.
API | Scripting | Description |
---|---|---|
- |
Performs ray-casting on the geometry of a 3D model, in order to produce Raster Data. |
Another way to process model data is to feed it into other parts of the Tinman 3D SDK. Here is a list of possible API calls:
-
For IModel objects:
-
For IModelReader objects:
-
For IModelTexture objects:
Raster Data
Raster data is primarily used to generate vertices for building terrain meshes. Please refer to Terrain Overview for more information.
Import
Importing raster data means to wrap a raster data source in a IHeightmap object. This step is often preceded with a pixel data processing step or a shape data processing step, which has produced the source raster data. Using Procedural Data, raster data may also be generated at runtime.
Please refer to the GeodataPipeline class as an example for a simple processing pipeline. |
API | Scripting | Description |
---|---|---|
Opens a dataset in read-only or read/write mode. |
||
Creates a heightmap with constant sample values. |
||
Creates a heightmap dataset. Storage space is only used for actual data. |
||
Creates an in-memory heightmap. Memory is only used for actual data. |
||
Uses Procedural Data to generate samples at runtime. |
||
- |
Creates a proxy heightmap which allows to swap the heightmap content later, by choosing another heightmap to wrap (see IHeightmapProxy.Content). |
|
Creates a heightmap that generates samples at runtime, by performing rasterization of map shapes (see IHeightmapShape.ShapeAdd). |
Consume
Consuming raster data means to assemble IHeightmap objects at runtime in order to access or transfer heightmap sample data (see HeightmapSample, HeightmapSamples and HeightmapRegion).
API | Scripting | Description |
---|---|---|
Merges a heightmap with a constant data, in order to fill in missing heightmap layers. |
||
Copies samples from one heightmap to another while optionally applying a custom copy operation on the samples. |
||
IHeightmap.GetSample |
- |
Reads samples from a heightmap. |
Generates data for the material layer by applying color-keys to the texture layer. |
||
- |
Creates a HeightmapProjector object that can be used to transfer samples between two heightmaps while performing required conversions and resampling, taking into account geo-referencing information. |
|
Transfers samples from one heightmap to another in a background operation while performing required conversions and resampling, taking into account geo-referencing information. |
||
- |
Writes samples to a heightmap. |
|
Fits a rectangular heightmap in a square one while smoothing the borders with a given fall-off. |
||
Uses the coverage layer of the second heightmap to remove data from the first heightmap. |
||
Combines the data of two heightmap, using the specified combine operations. |
||
Merges the data of two heightmaps, taking into account the coverage layers. |
||
- |
Creates a placeholder heightmap that can have its content replaced later. |
|
Scales a heightmap to a different raster size. |
||
Extracts a single layer from a heightmap. |
||
Swaps material IDs in the material layer. |
||
Transforms the vertical coordinates of the heightmap (i.e. elevation and displacement). |
||
Uses a color-ramp to generate data for the texture layer, based on the elevation layer. |
||
- |
A helper object that automatically merges heightmaps of different resolutions. |
Process
Processing raster data means to transform it to other data (e.g. terrain brushes, mesh chunks or pyramid data) or to use it for real-time terrain rendering.
API | Scripting | Description |
---|---|---|
- |
A terrain painting brush that sources data from a heightmap. |
|
- |
Uses the heightmap for geocentric computations that involve gravity-related height or height above surface. |
|
Reads a rectangular region of a heightmap as pixel data. |
||
Generates pyramid data for the given heightmap. |
||
- |
Uses the heightmap for building 3D models for selected terrain sectors. |
|
- |
Uses the heightmap for a terrain in the Low-level Terrain API or the High-level Terrain API. |
|
- |
Uses the heightmap for a terrain in the Scene API. |
Another way to process raster data is to feed it into other parts of the Tinman 3D SDK. Here is a list of possible API calls:
Pyramid Data
Pyramid data is primarily used to generate textures for terrain meshes. Please refer to Terrain Overview for more information.
Import
Importing pyramid data means to obtain a IPixelPyramid object that wraps some pyramid data source, in order to be able to access the contained pyramid tiles by reading and/or writing ColorBuffer data.
API | Scripting | Description |
---|---|---|
Streams BingMaps data. |
||
Creates a pyramid that wraps the given pixel data. |
||
- |
Creates a pyramid that counts accesses to pyramid tiles, which is useful to track billable transactions for online services. |
|
Streams GoogleMaps data. |
||
Creates a pyramid that shows regions for which raster data is available. |
||
Opens an |
||
Streams OpenStreetMaps data. |
||
Opens a dataset in read-only or read/write mode. |
||
Creates a pyramid with constant pixel values. |
||
Creates a pyramid dataset. Storage space is only used for actual data. |
||
Creates an in-memory pyramid. Memory is only used for actual data. |
||
Creates a pyramid that generates samples at runtime, by performing rasterization of map shapes (see IPixelPyramidShape.ShapeAdd). |
||
Accesses a web-service that delivers pyramid data, for example WMS or WMTS. |
Consume
Consuming pyramid data means to assemble IPixelPyramid objects at runtime in order to access or transfer pyramid data (see ColorBuffer).
API | Scripting | Description |
---|---|---|
Wraps a pyramid in an in-memory cache or a file-based cache. |
||
Combines the data of two pyramids, using the specified combine operations. |
||
Copies pyramid tiles from one pyramid to another one. |
||
Computes pixel data for empty pyramid tiles by downsampling the four child tiles. |
||
- |
Aggregates another pixel pyramid and performs custom drawing on the pixel data on its pyramid tiles. |
|
- |
Reads pixel data for the given pyramid tile. |
|
- |
Writes pixel data for the given pyramid tile. |
|
Slices a pyramid so that pixel data is returned only for a given range of pyramid levels. |
||
Scales a pyramid to the given full size (i.e. the size of the bottom-most level). |
||
Specifies the amount of tile padding for the pyramid. |
||
Specifies the tile size of the pyramid. |
||
Performs re-projection of pixel data in order to convert a rectangular pyramid to a cubemap pyramid. |
||
- |
A helper object that automatically merges pyramids of different resolutions. |
Process
Processing pyramid data means to transform it to other data (e.g. texture data or mesh chunks) or to use it for real-time terrain rendering. The IPixelPyramid and ITexelPyramid interfaces both represent pyramid data: the former provides data as ColorBuffer objects, whereas the latter provides data as TexelBuffer objects.
API | Scripting | Description |
---|---|---|
Encodes a pixel pyramid into a texel pyramid. |
||
Reads a rectangular region of a pyramid as pixel data. |
||
Wraps a texel pyramid in a file-based cache. |
||
- |
Decodes a texel pyramid into a pixel pyramid. |
|
- |
Uses a texel pyramid for 2D map rendering. |
|
- |
Uses a pixel pyramid to generate textures for 3D models of terrain mesh chunks. |
|
- |
Uses a pixel pyramid for real-time 3D terrain rendering. |
|
- |
Uses a texel pyramid to provide texture data for a GPU-based texture atlas. |
|
- |
Uses a pixel pyramid as a texture layer in a scene. |
Another way to process pyramid data is to feed it into other parts of the Tinman 3D SDK. Here is a list of possible API calls: