t-rex-tileserver/t-rex

t-rex is a vector tile server specialized on publishing MVT tiles

Vector tile publishing using your own data

t-rex

t-rex is a vector tile server specialized on publishing MVT tiles from your own data.

Features

  • Support for PostGIS databases and GDAL vector formats
  • Auto-detection of layers in data source
  • Built-in viewers for data display and inspection
  • Tile generation command with simple parallelization
  • Automatic reprojection to grid CRS
  • Support for custom tile grids

Presentations

  • T-rex, a vector tile server for your own data (FOSS4G 2017): slides
  • Vector Tiles - Introduction & Usage with QGIS (User meeting Bern 21.6.17): slides
  • Von WMS zu WMTS zu Vektor-Tiles (FOSSGIS 2017): Video
  • Workshop "Vector Tiles" (GEOSummit Bern 7.6.16): slides

Examples

  • AdV Smart Mapping
  • Swiss Ornithological Institute, Birds of Switzerland

Usage

  • Setup
  • Serving vector tiles
  • Generating vector tiles
  • Configuration

Quick tour

t_rex serve --dbconn postgresql://user:pass@localhost/osm2vectortiles

Tiles are then served at http://localhost:6767/{layer}/{z}/{x}/{y}.pbf

A list of all detected layers is available at http://localhost:6767/

Use a tile cache:

t_rex serve --dbconn postgresql://user:pass@localhost/osm2vectortiles --cache /tmp/mvtcache

Generate a configuration template:

t_rex genconfig --dbconn postgresql://user:pass@localhost/osm2vectortiles | tee osm2vectortiles.toml

Run server with configuration file:

t_rex serve --config osm2vectortiles.toml

Generate tiles for cache:

t_rex generate --config osm2vectortiles.toml

Increase log level:

t_rex serve --loglevel debug --dbconn postgresql://user:pass@localhost/osm2vectortiles

For developers

t-rex is written in Rust. Minimal required rustc version is 1.45.

Software Requirements

Ubuntu 20.04 (Focal Fossa):

sudo apt install cargo libssl-dev libgdal-dev

Build and run

Build:

cargo build

Run tests:

cargo test --all

Run server with DB connection:

cargo run -- serve --dbconn #404

Decode a vector tile:

curl --silent http://127.0.0.1:6767/ne_10m_populated_places/5/31/17.pbf | protoc --decode=vector_tile.Tile t-rex-core/src/mvt/vector_tile.proto

Database tests

Unit tests which need a PostgreSQL connection are ignored by default.

Start Test DB:

docker run -p 127.0.0.1:5439:5432 -d --name trextestdb --rm sourcepole/trextestdb

To run the database tests, declare the connection in an environment variable DBCONN:

export DBCONN=#404

Run the tests with

cargo test --all -- --ignored

Creating test database locally:

# Set Postgresql environment variables when needed: PGHOST, PGPORT, PGUSER, PGPASSWORD
cd data
make createdb loaddata

S3 tests

Unit tests which need a S3 connection are skipped by default.

Install MinIO Client.

Start Test S3

docker run -d --rm -p 9000:9000 -e MINIO_REGION_NAME=my-region -e MINIO_ACCESS_KEY=miniostorage -e MINIO_SECRET_KEY=miniostorage minio/minio server /data && sleep 5 && mc config host add local-docker http://localhost:9000 miniostorage miniostorage && mc mb local-docker/trex && mc policy set download local-docker/trex 

To run the S3 tests, declare that there is a S3 available in an environment vaiable S3TEST:

export S3TEST=true

Run the tests with

cargo test --all -- --ignored

License

t-rex is released under the MIT License.

Issues

Collection of the latest Issues

joto

joto

Comment Icon0

The documentation says I can run t_rex serve with --cache DIR to enable the cache. But this doesn't work with last released version. In the log it says somewhere "No cache". I can enable the cache in the config file, but I'd rather do this on the command line, because it is more flexible that way.

gozell

gozell

Comment Icon0

hello

with simplify = false we have found some corrupted polygon areas using t-rex v0.14.0, v0.14.1, v0.14.2, v0.14.3, and v0.13.0 with clipping tile borders.

For example this corrupted polygon area with simplify = false: ** buffer_size = 30 simplify = false** https://maputnik.github.io/editor/?style=https://dev.adv-smart.de/styles/playground/gold/bm_web_col_3857_errTest_Schwarzwald.json#15.01/48.67958/8.23535 grafik

with a slight simplify the result is correct: buffer_size = 30 simplify = true tolerance = "!pixel_width!/15" https://maputnik.github.io/editor/?style=https://dev.adv-smart.de/styles/playground/gold/bm_web_col_3857_errTest.json#15.01/48.67958/8.23535 grafik

Perhaps there is a connnection with issue 238 ? https://github.com/t-rex-tileserver/t-rex/issues/238

Look at #404 for the complete map: https://dev.adv-smart.de/map-view/e01ba67e-b4a4-4acf-ac58-866d8f215bff

roelarents

roelarents

Comment Icon3

When serving tiles from a geopackage (~85 GiB, ~20 layers) I see that the response times are drastically lower when requests are made in parallel. E.g. when I create test requests for 6 tiles in parallel the response time is ~8000ms versus ~300ms when I create them one after another.

I've tried looking at the source code (I'm a Rust rookie) but I couldn't find a hint yet. I tried. I've tried supplying the NOLOCK open option. Thinking that it might be the overhead of a locked GPKG that is trying to be opened by multiple clients at the same time. Even though it should by possible to open an sqlite db concurrently. That helped a little to get it to ~7000 ms.

Does anyone have an idea where the delay in response times might originate?

Edit: connection is already opened read-only by default. Opening shared would be unsafe. Edit2: Made an image with gdal 3.4.2 and t-rex to try out NOLOCK. Edit3: The same effect happens with a FlatGeoBuf datasource.

ChristianBeilschmidt

ChristianBeilschmidt

Comment Icon1

I'm using T-Rex to generate WGS84 vector tiles for Natural Earth Data. For this, I import the geopackage into PostGIS and then use this as a datasource for T-Rex.

One layer is specified like this:

It seems to work in general :tada: but unfortunately every line gets simplified to having only ~3 nodes.

grafik

Then I found the tolerance parameter, which is per default !pixel_width! / 2. However, there was no difference when I changed the 2 to 4, 16 or 100 – which appeared very strange.

I look into the codebase about the database queries and the string replacement seems to be working just fine, so this left me to question what pixel_width is. In more detail, I found the following code:

https://github.com/t-rex-tileserver/t-rex/blob/f2433f648eee4d793e076e1a7570ba7666b4b232/tile-grid/src/grid.rs#L185-L193

So pixel width has something to do with 256-pixel tile width in meters for Web Mercator.

Then, I tried to divide the pixel width with the fixed value that was multiplied to our degree values. This leads to the following setting:

And this seems to solve the simplification problem to reasonable country borders.

grafik

So I guess either there should either be different defaults for different projections for tolerance or the pixel size for degrees is debatable.

yarray

yarray

question
Comment Icon3

I tried multiple sources including Natural Earth, UIA, etc., with both shapefile and postgis as datasources. However, although I could see some lines/polygons sometimes, it seemed that the tiles were always incorrect. To illustrate the problem, I created two datasets of polygons.

The first one contained several large polygons, like:

截屏2022-02-16 下午1 14 50

I then generated config with

./t_rex genconfig --datasource <path>/large_polygon.shp > config.test.toml

and then served the tileset with

RUST_LOG=debug ./t_rex serve -c config.dresden.toml

In the Xray tab:

截屏2022-02-16 下午1 28 10

The log printed "1 features" for some tiles. It also occasionally printed "ERROR 1: latitude or longitude exceeded limits". I'm not sure whether it's related to the final output.

截屏2022-02-16 下午2 39 10

The corresponding config was:

The input shapefile was large_polygon.zip.

sanderpukk

sanderpukk

question
Comment Icon1

Hi!

Generating tiles in cloudservices. For this example, let's say it's Azure's 72 node and 144 GB of RAM machine.

First the performance. The documentation is a little lacking in that part. We are using Postgres as a source. Now, how would I approach the needed nodes for t-rex and the nodes for postgres?

  • Am I supposed to use (72/2)-1 nodes for t-rex and the rest for postgres?
  • What does that "pool" parameter actually do in your documentation?
  • Besides functionality, is Postgres connection the most optimal? Are GPKG better for performance?

A better documentation for performance would be much appreciated.

Second point: When using custom grid and extent and extent in tileset, it does clip accordingly. But when using built in web_mercator grid, the extent is confusing.

Tileset extent in your github examples is an array of coordinates. BottomLeft to RightTop. But in the webpage documentation it's a json extent: { minx: float, miny: float, maxx: float, maxy: float }. Same as for user grid.

The thing is, if I use grid.user with my extent and the same value in tileset, it does clip. Webmercator doesnt seem to work like that. What is the correct way?

FreyTalking

FreyTalking

Comment Icon0

Hi there, I've been using this tool for quiet some now and it worked very well mostly. I was able to use the Web Mercator projection and my own tile schemes (UTM32WGS84Quad, AdV_25832 and gdi_de_25832). When I proceeded to use the WGS84 projection I got this error and was unable to fix it:

grafik

The viewer I use is based upon OpenLayers. Has anybody an idea, where I might be mistaking? Many thanks in advance.

pvgenuchten

pvgenuchten

Comment Icon1

I tried replacing favicon.ico by placing it in a folder /tmp then in config reference the static folder as /tmp and path / favicon gets correctly replaced, but a side-effect is that any other path outside static, returns 404:

  • localhost:6767/ renders 404 not found, but localhost:6767/index.html works fine
  • localhost:6767/config.json renders 404
maxammann

maxammann

Comment Icon8

Hello, I'm using the tile-grid library outside of t-rex for determining Slippy map tilenames.

I'm a bit confused about the values which are returned by the library. For example:

  1. On zoom level 0, there is only a single tile, which covers the whole world. This means that the extend in WSG84 of the tile z=0,x=0,y=0 should be:

But Grid::wgs84().tile_extent_xyz(0, 0, 0) returns Extent { minx: -180.0, miny: -90.0, maxx: 0.0, maxy: 90.0 }. I belive this is not correct or at least I do not understand the result. The implementation in mapcache is equal: https://github.com/mapserver/mapcache/blob/main/lib/grid.c#L51

  1. I want to iterate over a few map tiles in Germany. Therefore, I'm doing:

If I fetch the tiles which have been collected, then I'm not able to fetch them from my hosted tile server (nginx with pbf files): https://maps.tuerantuer.org/europe_germany/tiles.json

elariroop

elariroop

Comment Icon0

Version 0.14.2

It seems that the support for GeometryCollection is not complete:

2021-11-24 17:03:39.663 INFO GET /layer/6/33/35.pbf HTTP/1.1 500 0 926.654621ms thread 'actix-web' panicked at 'Unsupported geometry type 7 (Geometry Collection)', t-rex-gdal/src/gdal_fields.rs:321:26

elariroop

elariroop

Comment Icon2

Is there a way to parse datetime in field?

2021-11-24 15:30:20.209 WARN Layer 'layer' - skipping field 'valid_from': UnhandledFieldType { field_type: 11, method_name: "OGR_Fld_GetType" } 2021-11-24 15:30:20.209 WARN Layer 'layer' - skipping field 'created_date': UnhandledFieldType { field_type: 11, method_name: "OGR_Fld_GetType" }

Field info through ogrinfo: created_date (DateTime) = 2021/06/23 08:52:56.801

dmitrykinakh

dmitrykinakh

question
Comment Icon5

Hi there,

in the project we are building at the moment the biggest problem is tiles generation speed.

So, these are input parameters:

T-Rex version 0.14.1

Config in toml.

Number of records in DB - 13231 Number of records that match query in selected bbox - 60

Generation logs for:

Timing

Generation was performed on ec2 r5.4xlarge (16 vCPU 128 GB RAM) and with DB db.r6g.4xlarge (16 vCPU 128 GB RAM)

With bigger zoom levels, like 17 or 18 this takes hours. We are wondering if the issue is in data that we store as multipolygons. Those multipolygons contain from 1 to 2-3K individual polygons. image

From the performance perspective, would it be better to have these data represented in DB as polygons, so instead of 60 records, in our case, we will have several thousand? Looking forward to recommendations.

Note: this is a part of the query that takes the most of the time:

image

dmitrykinakh

dmitrykinakh

Comment Icon0

We are using https://github.com/t-rex-tileserver/t-rex/releases/tag/v0.14.1 version to do tiling. However, we have noticed that on some zoom levels. For this particular layer, it happens only on zoom level 11 (I've tried to generate up to z16)

Not sure that the problem is somewhere in data since in that case, that should be a problem on all zoom levels and not on a specific one. Looking for any recommendation on what to check in configs to prevent this.

musicformellons

musicformellons

question
Comment Icon1

I'm trying to have a docker container on Mac by following along the docs:

But keep getting Error reading configuration - Could not find config file!

I also tried copying my config file into the container but error remains. What am I doing wrong?

vladimir-rybalko

vladimir-rybalko

question
Comment Icon2

Hi!

I try configure t_rex for display more than 7k points from PG. Each user have uid in database. I want filtering data that belongs to the user. How can I pass a parameter to filter data in a SQL query?

tobwen

tobwen

Comment Icon1

What?

Starting with PostGIS v3.2, ST_AsMVTGeom() preserves more of the original geometry's details at a scale close to the target extent. If you need previous simplifying behavior, you need to simplify the geometry in advance.

But this is still in discussion as of writing (2021-10-01), since there might be performance problems.

References

https://git.osgeo.org/gitea/postgis/postgis/raw/tag/3.2.0alpha1/NEWS https://trac.osgeo.org/postgis/ticket/4149 https://github.com/postgis/postgis/pull/463

joto

joto

enhancement
Comment Icon2

I see that statistics collection is already built into t-rex, but it is only exposed in the drilldown command. I would like to get the statistics also in normal production so that I can continually track stats and see when there are regressions after changes etc.

I am not sure yet what the best format is. For the time being maybe having an extra (csv) log file that logs the stats would be okay, medium term we could add fancy things like putting the stats into special databases or whatever.

musicformellons

musicformellons

Comment Icon9

When trying to use the docker version I get:

Would it be possible to have an image for ARM64 ?

tobwen

tobwen

enhancement
Comment Icon1

background

I'm running NGINX as a remote proxy for t-rex on endpoint example.org/mvt/. I want to be able to have access to the "debug view" of t-rex, which is protected by HTTP Basic authentication.

issue

It seems like t-rex is designed to work on the root endpoint only (example.org/). All the static websites are linked to root, f.e. https://github.com/t-rex-tileserver/t-rex/blob/d5a67c47a1e2cf21a29b63bf820ed749f1cad911/t-rex-webserver/src/static/index.html#L1

As you can see, index.html always tries to read the JS-files from example.org/ instead of example.org/mvt/ - so normal rewriting the URL doesn't help.

what I've tried

  1. Of course, I'm aware of https://github.com/t-rex-tileserver/t-rex/issues/83, but none of the headers apply to index.html.
  2. I tried to use NGINX substitutions (https://nginx.org/en/docs/http/ngx_http_sub_module.html), but it isn't able to read HTML content from t-rex, seems like it's compressed?
  3. I then modified index.html and friends and put it into the static directory. It loaded up, but the layers haven't been loaded. It all resulted to NULL only. Seems like static cannot be used for this.

suggestions

MapProxy and other applications listen to the X-Script-Name and take care of this. With https://github.com/t-rex-tileserver/t-rex/issues/83 you actually have similar abilities, but for the JSON files, not for the "debug view". Could you add this? It would be really nice to have a realtime look into the data, even when it's running on a non-root location.

Vangogh-dev

Vangogh-dev

question
Comment Icon3

Hello,

I am working on a project that needs a vector tile map. I am using postgresql with geofabrik data imported with osm2pgsql and t-rex to serve it. I can display data from my db but the goal is to display a basemap to load geojson on top. What is the next step to get my basemap ? I am a bit lost in the web map/geographic data community because i am just begining. t_rex_alsace_roads Here is what t-rex is displaying from the db

PS : one of my late goal is to display 3d building so if you have solutions or info about that i would love to ear it.

xycarto

xycarto

Comment Icon2

This issue builds off of #213

Building a tile cache custom projection from a custom projection works great. See for results of an elevation contour in NZTM: https://xycarto.github.io/elevation.viewer.test/

The new issue I seem to be facing is that the elevation data is not properly populating the metadata.json file.

This is the output:

https://github.com/xycarto/elevation.viewer.test/blob/main/wellycontour_simple/metadata.json

Maybe I am reading this file incorrectly, but should I not be getting the elevation values returned as well? Not just the attribute name?

dmitrykinakh

dmitrykinakh

enhancement
Comment Icon3

Hi @pka,

I'm interested if Trex supports passing variables to the style.json file into the filters section.

For example: Right now in the project were we display vector tiles we do that per city so to display Miami and North Miami we need to create two style files. Smth, like:

  • fl-state-place-1245000.style.json
  • fl-state-place-1249450.style.json

Inside each of those files, we specify filter right now: image

We are looking for a way how to optimize that and have only 1 file (in reality we support ~30 cities right now), so there will be like one file called fl-state-place-{placeholder of city id here}.style.json and as a result, {placeholder of city id here} will be applied for the filter in section I've mentioned above. Is this possible?

luqinghui

luqinghui

enhancement
Comment Icon5

Step1

I download some data from GADM, and i import it to postgis using shp2pgsql.

So now I have a layer(afg).

Step2

I transfrom the data to 3857 using the st_transfrom(geom, 3857).

So now, I have two layers(afg and afg_3857).

Step3

I generate some point in afg polygon using the st_generatepoints(geom, 2000).

So now I have three layers(afg, afg_3857 and afg_point_4326).

Step4

I generate the config file using t_rex genconfig. Then I change afg_3857 srid to 3857 and afg_point_4326 to 4326(the original is 0, they cannot be auto recongized correctly).

Step5

When I start the server, the afg_3857 and afg_point_4326 can be displayed by viewer, but the original data afg can not. 图片 图片 图片

Why the same srid(4326), one can display but another can not? I am confused if my config file is wrong or it is a bug or feature? My system is Debian 9 and t_rex version is v0.8.2(i tried v0.10.0, the result was same as v0.8.2).

silverwand

silverwand

enhancement
Comment Icon2

I wonder if there is a way to set t-rex to work in https mode (on port 443 or on any other port) ? I didn't find any information about it in the documentation .

Any idea ?

Thanks a lot.

trip1

trip1

Comment Icon3

First, I really enjoy working with the t-rex server, runs fast and really useful in generating our maps.

Only issue I seem to be having is with one gpkg file of the oceans. I'm taking the water polygon shape file from osm coastline and using ogr2ogr ogr2ogr -f GPKG water.gpkg water_polygons.shp to convert to gpkg files.

When inspecting both the shapefile and the gpkg file in qgis both render correctly with no issue. Not sure where I might have messed something up or if there's an issue rendering gpkg files with the full ocean polygon. Here's an example of the issue in rendering.

Screenshot from 2019-07-11 08-28-20

Here's my test config file:

boldtrn

boldtrn

Comment Icon6

I was wondering if there is a getting started like tutorial to use t-rex with OSM?

If I understand the docs correctly, it is enough to setup a Postgis DB with e.g. osm2pgsql or imposm? Is there a config file that creates a well used vector tile schema like Mapbox, OpenMaptTiles Schema, or maybe plain OSM Schema?

For example I do have a .tm2source from OpenMapTiles. I think t-rex can't read tm2sources but I was wondering if the way to go would be to try and convert the tm2source?

Versions

Find the latest versions by id

v0.14.3 - Jan 29, 2022

See CHANGELOG

v0.14.3-beta4 - Dec 09, 2021

See CHANGELOG

v0.14.3-beta3 - Dec 09, 2021

See CHANGELOG

v0.14.3-beta2 - Nov 24, 2021

See CHANGELOG

v0.14.3-beta1 - Nov 24, 2021

See CHANGELOG

v0.14.2 - Nov 11, 2021

See CHANGELOG

v0.14.1 - Oct 04, 2021

See CHANGELOG

v0.14.1-pre9 - Oct 01, 2021

See CHANGELOG

v0.14.1-pre6 - Sep 22, 2021

See CHANGELOG

v0.14.0 - Aug 27, 2021

See CHANGELOG

v0.13.0 - Feb 19, 2021

See CHANGELOG

v0.12.0 - Dec 26, 2020

See CHANGELOG

v0.12.0-alpha5 - Dec 26, 2020

See CHANGELOG

v0.12.0-alpha1 - Dec 21, 2020

See CHANGELOG

v0.11.0 - Jun 25, 2020

See CHANGELOG

v0.10.1 - Feb 20, 2020

See CHANGELOG

v0.10.0 - Aug 04, 2019

See CHANGELOG

v0.9.3 - Jun 09, 2019

See CHANGELOG

v0.9.2 - Apr 27, 2019

See CHANGELOG

v0.9.1 - Nov 14, 2018

See CHANGELOG

v0.9.0 - Jul 25, 2018

See CHANGELOG

v0.8.2 - Apr 18, 2018

See CHANGELOG

v0.8.1 - Oct 05, 2017

See CHANGELOG

v0.8.0 - Sep 26, 2017

See CHANGELOG

v0.8.0-beta1 - Sep 20, 2017

v0.8.0-pre2 - Sep 02, 2017

v0.8.0-pre1 - Aug 15, 2017

v0.7.8 - Aug 05, 2017

See CHANGELOG

v0.7.7 - Jul 14, 2017

See CHANGELOG

v0.7.6 - Jul 10, 2017

See CHANGELOG

Information - Updated Sep 10, 2022

Stars: 462
Forks: 50
Issues: 46

Rust DataBase Connectivity (RDBC)

Love them or hate them, the JDBC standards have made it easy to use a wide range of desktop and server products with many different...

Rust DataBase Connectivity (RDBC)

Powerful database anonymizer with flexible rules

See what else we English and

Powerful database anonymizer with flexible rules

A files database for nixpkgs

nix-index is a tool to quickly locate the package providing a certain file in checkout the latest here) or use Nixpkgs' repositories' and install it...

A files database for nixpkgs

Memory based database with integrated Web Socket Server

Takes around 30µs (microseconds) to locate a specific account from amongst 50 million other accounts

Memory based database with integrated Web Socket Server

Simple Database Implementation written in Rust

This repository uses cargo 1

Simple Database Implementation written in Rust

Effortless database migrations for SeaORM!

Effortless database migrations for

Effortless database migrations for SeaORM!

tarkov-database-rs

Rust library provides low level bindings for the

tarkov-database-rs

A database-backed logger for use with the log crate

db_logger is a Rust crate providing an implementation of the

A database-backed logger for use with the log crate

Local database solution for sit

uations that requires strict data integrity and absolute portability

Local database solution for sit
Facebook Instagram Twitter GitHub Dribbble
Privacy