Streaming and Caching

This page explains how to stream data from remote content with the Filesystem and Networking abstraction layers and how to configure and use the data caches.

Streaming

This section shows how to stream remote content.

Remote Dataset Files

Raster datasets (*.hgt) and pyramid datasets (*.pyr) are plain files that contain an embedded database (see BlockStorage).

To refer to a dataset file, Path objects are used (see the Filesystem abstraction layer for details). Usually, path values refer to files in the local filesystem, but they may also point to other filesystems, for example a remote HTTP/HTTPS filesystem (see Remote Files).

To stream a raster of pyramid dataset file, it it sufficient to upload it onto a regular web server that supports HTTP/1.1 byte range GET requests. Then, the uploaded dataset file can be used similarly to a local one. The binary dataset layout has been designed so that it well supports incremental streaming via network communication.

To avoid redundant streaming of remote dataset content, a Dataset Cache may be used, which will cache the downloaded data in a local file, so that it may be used again the next time the application runs.

Remote Web Sources

On the web, pyramid data is commonly provided via HTTP/HTTPS endpoints that deliver pyramid tile images, for example as PNG or JPEG files. To access such a web service, an UrlPatternPyramid object may be used, which involves the following steps:

  1. Provide the geo-reference of the web service by building a Raster object.

  2. Provide the URL pattern for accessing the web service, by building a Label object.

  3. Create the UrlPatternPyramid object.

If the URL pattern mechanism is not flexible enough, derive a custom class from HttpPyramid and implement the GetTileUrl method. Please refer to the BingMapsPyramid, GoogleMapsPyramid and OpenStreetMapPyramid classes as examples.

To avoid redundant requests to remote web sources, a Pyramid Cache may be used, which will cache the downloaded data in a local file, so that it may be used again the next time the application runs.

Custom Web Service

This is an example for streaming pyramid data from a custom non-WMS / non-WMTS web service.

See also

Example_PixelPyramid.StreamFromCustomService.

Digitized Sky Survey / AAS WorldWide Telescope
CoordinateOperation coordOp;
CoordinateSystem coordSys;
double scale;
int center;
IRasterTransform transform;
Raster georef;
Label label;

// Setup geo-reference.
coordOp = new CoordinateOperation(CoordinateOperationMethod.Toast);
coordOp.Set(CoordinateOperationParameter.Scale, -6);
coordSys = new CoordinateSystem(new GeodeticDatum(
  new Ellipsoid(1000000), new PrimeMeridian(0)), coordOp);
scale = 180.0 / MappingUtil.MaxSize;
center = MappingUtil.MaxSize / 2;
transform = RasterTransform.TiepointScale(new Vec2D(center, center),
  Vec2D.Zero, new Vec2D(scale, -scale));
georef = new Raster(coordSys, transform);

// Create pixel pyramid.
label = Label.From(
  "https://www.worldwidetelescope.org/wwtweb/dss.aspx?q={0},{1},{2}"
);
return new UrlPatternPyramid(label, 256, 12,
  UrlPatternMode.Tiled1x1, PyramidFlags.Opaque, georef);

Web Map Service (WMS)

This is an example for streaming pyramid data from a Web Map Service (WMS).

See also

Example_PixelPyramid.StreamFromWebMapService.

NASA Earth Observations / Land Cover Classification (1 year)
StringBuilder pattern;
double scale;
Raster georef;
Label label;

pattern = new StringBuilder();

// Set the base URL of the service.
pattern.Append("https://neo.gsfc.nasa.gov/wms/wms");

// Set common request parameters.
pattern.Append(
  "?REQUEST=GetMap&VERSION=1.3&CRS=CRS:84&FORMAT=image/png");

// Choose the data layer.
pattern.Append("&LAYERS=MCD12C1_T1");

// Provide mapping from pyramid tile parameters to WMS parameters.
pattern.Append("&BBOX={1},{4},{3},{2}&HEIGHT={0}&WIDTH={0}");

// Setup geo-reference.
scale = 360.0 / MappingUtil.MaxSize;
georef = new Raster(null, RasterTransform.TiepointScale(
  Vec2D.Zero, new Vec2D(-180, 90), new Vec2D(scale, -scale)));

// Create pixel pyramid.
label = Label.From(pattern.ToString());
return new UrlPatternPyramid(label, 256, 8,
  UrlPatternMode.Contiguous, PyramidFlags.Opaque, georef);

Web Map Tile Service (WMTS)

This is an example for streaming pyramid data from a Web Map Tile Service (WMTS).

See also

Example_PixelPyramid.StreamFromWebMapTileService.

NASA Worldview / Earth at Night 2012
StringBuilder pattern;
double scale;
Raster georef;
Label label;

pattern = new StringBuilder();

// Set the base URL of the service.
pattern.Append(
  "https://map1a.vis.earthdata.nasa.gov/wmts-geo/wmts.cgi");

// Set common request parameters.
pattern.Append(
  "?SERVICE=WMTS&VERSION=1.0.0&REQUEST=GetTile&FORMAT=image/jpeg");

// Choose the data layer.
pattern.Append(
  "&LAYER=VIIRS_CityLights_2012&TILEMATRIXSET=EPSG4326_500m");

// Provide mapping from pyramid tile parameters to WMS parameters.
pattern.Append("&TILEMATRIX={0}&TILECOL={1}&TILEROW={2}");

// Setup geo-reference.
scale = 360.0 / 0.625 / MappingUtil.MaxSize;
georef = new Raster(null, RasterTransform.TiepointScale(Vec2D.Zero,
  new Vec2D(-180, 90), new Vec2D(scale, -scale)));

// Create pixel pyramid.
label = Label.From(pattern.ToString());
return new UrlPatternPyramid(label, 512, 9,
  UrlPatternMode.TiledNon1x1, PyramidFlags.Opaque, georef);

Remote Files

The Filesystem abstraction layer is used to access files via Path objects. By using the path selectors http and https, paths may refer to the built-in HttpFileSystem, which will produce HttpFile objects.

Use of remote files is handled transparently by the High-level Terrain API and Scene API
Open a remote HTTPS file in read-only mode
string url;
Path path;
IFile file; // [!]

url = "https://data.tinman3d.online/geo/Earth/EGM96/egm96.hgt";
path = Path.From(url);
file = path.FileNew(FileFlags.Read);

The IFile.Read method of HttpFile objects use the ISimpleHttp.GetBytes method, which performs HTTP/1.1 GET requests. Instead of using the default SimpleHttp object, client-code may specify a custom object.

Use a custom ISimpleHttp object for the HttpFileSystem
ISimpleHttp http; // [!]

http = new SimpleHttp("MY-OWN-AGENT");

HttpFileSystem.Instance.Http = http;

By default, a HTTP/HTTP Get request is made for each call to IFile.Read, which might be sub-optimal. To avoid unnecessary requests, a File Cache may be used.

I/O Caching

This sections describes the caches that are intended to reduce I/O work, by keeping most-recently used data in memory.

Dataset Cache

The dataset cache holds the most recently used LOD blocks of raster datasets and/or the most recently used tiles of pyramid datasets in a local file (usually named *.hgc). For each dataset, the cached data is stored separately, according to the unique dataset identifier. In combination with Remote Dataset Files, this can be used to avoid redundant network communication.

To use the dataset cache, a DatasetFileCache object must be created and associated with individual raster and pyramid datasets by using the IDataset.UseCache method.

Use a dataset file cache for a dataset.
DatasetFileCache cache; // [!]
IHeightmapDataset dataset; // [!]

// Open dataset.
dataset = Heightmap.Datasets.Open("dataset.hgt");

// Create dataset file cache.
cache = DatasetFileCache.Create(Path.From("cache.hgc"));

// Associate file cache with dataset.
dataset.UseCache(cache);

File Cache

To use general-purpose file caching, a FileCache object must be created and associated with individual files using the IFileOps.Cache method.

Use a FileCache for a read-only file.
IFile file; // [!]
FileCache cache; // [!]

// Open file in read-only mode.
file = Path.From("file.dat").FileNew(FileFlags.Read);

// Create file cache.
cache = new FileCache(16777216);

// Associate file cache with file.
file = file.Cache(cache);

When file caching is used, the file content is divided into equal chunks of data (i.e. cache pages), which are loaded lazily upon first request and kept in the memory cache, using a least-recently used eviction scheme.

Pyramid Cache

The pyramid cache is intended to keep commonly used pyramid tiles in local files (usually named *.pyc), in order to avoid unnecessary streaming of Remote Dataset Files or execution geodata processing tasks.

To use the pyramid cache, a PyramidFileCache object must be created and associated with individual pixel or texel pyramids by using the IPixelPyramid.Cache resp. ITexelPyramid.Cache method.

Unlike the Dataset Cache, the application is responsible for providing unique storage identifiers for each pyramid that is associated with the pyramid cache.

Use a PyramidFileCache for a pixel pyramid.
PyramidFileCache cache; // [!]
IPixelPyramid pyramid; // [!]
int pyramidId;

// Create pixel pyramid.
pyramid = new OpenStreetMapPyramid();

// Create pyramid file cache and choose storage identifier.
cache = PyramidFileCache.Create(Path.From("cache.pyc"));
pyramidId = 1;

// Associate pyramid cache with pixel pyramid.
pyramid = pyramid.Cache(cache, pyramidId);

CPU Caching

This section describes the available caches for CPU-based terrain data processing. Each of these caches has a configurable memory size. The caches will be created upon first use and will be disposed when no longer used.

To avoid that caches are created and disposed repeatedly because of sporadic use, the following properties may be used:

Database Cache

The database cache is used by the BlockStorage and BlockIndex classes, in order to manage B-tree nodes efficiently. It does not store actual data.

The size of the database cache can be fine-tuned with the DatabaseUtil.CacheMemory property.

Usually, the database cache does not need to be fine-tuned. Tuning might be advantageous for applications that perform intense geodata processing.

The following APIs make use of the BlockStorage class and are thus affected by the database cache:

Heightmap Cache

The heightmap cache is used to keep most-recently used raster data in memory.

The size of the heightmap cache can be fine-tuned with the HeightmapsUtil.CacheMemory property.

The default heightmap cache size is a reasonable general-purpose value that depends on the total amount of memory.

The following APIs make use of the heightmap cache:

Pyramid Cache

The pyramid cache is used to keep most-recently used pyramid data in memory.

The size of the pyramid cache can be fine-tuned with the PyramidsUtil.CacheMemory property.

The default pyramid cache size is a reasonable general-purpose value that depends on the total amount of memory.

The following APIs make use of the pyramid cache:

GPU Caching

This section describes the available caches for GPU-based terrain data rendering.

Geometry / Mesh Data

For real-time 2D / 3D rendering of raster data, geometry data must be generated and uploaded to the GPU.

Based on raster samples, vertex data is generated and stored in vertex buffer resources on the GPU. For terrain meshes, these vertex buffers are used as a vertex data cache with a least-recently used eviction scheme.

During terrain refinement, a triangle mesh is generated. The structure of that mesh may be stored in custom buffer resources on the GPU. This information can be used to perform triangulation on the GPU.

In the Low-level Terrain API, the capacity of the vertex data cache may be specified by choosing the capacity when creating the VertexArrays objects, which must be passed to MeshBuffer.Create.

In the High-level Terrain API, the cache capacity may be specified via TerrainBufferOptions.VertexCapacity.

In the Scene API, the capacity of the vertex data cache is chosen automatically and cannot be configured from the outside.

Texture Data

For real-time 2D / 3D rendering of pyramid data, texture data must be generated and uploaded to the GPU.

Pixel data is usually generated by a IPixelPyramid object and then converted to texture data via EncodePixels. The resulting ITexelPyramid object is then associated with a TextureAtlas object, which provides a cache for texture tiles with a least-recently used eviction scheme.

Texture Format

Specifies the binary encoding of the texture data that will be uploaded to the GPU, see ITextureFactory.ValidateFormat. Depending on the intended use of the texture data, different texture formats may be chosen.

Texture Size

Specifies the width and height of each texture in the atlas and will always be a power of two, see ITextureFactory.ValidateTextureSize.

Texture Count

Specifies the number of textures that will be used to form the texture tile cache, which basically controls the capacity of the cache and is limited by the amount of available GPU memory, see ITextureFactory.AvailableVideoMemory.

Slice Count

Specifies the number of textures to pack into a single GPU texture array resource, see ITextureFactory.MaximumCount.

Tile Size

The tile size of a texture atlas is the size of the pyramid tiles that may be cached. If necessary, IPixelPyramid.ToTileSize will be used if a pixel pyramid uses a different tile size.

In the Low-level Terrain API, the TextureAtlas object must be created and configured explicitly, which allows full control over the texture data cache options.

In the High-level Terrain API, the texture data cache options must be must be applied to a TextureAtlasOptions object that is is returned by TerrainBufferOptions.TextureAtlas.

In the Scene API, texture data cache options must be specified when declaring a texture atlas via SceneOptions.TextureAtlas.