View Single Post
Posts: 1,414 | Thanked: 7,547 times | Joined on Aug 2016 @ Estonia
#415
New version of OSM Scout Server (1.4.0) and updated maps are out.

For those using Jolla Harbour, you can start using new maps only after Jolla Harbour version is updated as well.

Main work was done not on the server side itself, but on optimization of Mapbox GL tiles import, adjustments of the tile schema, and generation of map styles. On server side, I mainly updated documentation by adding sections on running the server and what does automatic activation means in terms of file generations at the home directory (asked during Jolla Harbour review). Thanks to the translators for updated strings, I included the version available this morning.

Now to give you an idea of what's going on with the implementation and import of the tiles, I describe it a bit below. While we use Mapbox GL Native as a library, lots of work done by Mapbox on styling is not available for use in external projects (when you want to use it independently from Mapbox service). Which is fine since they do have to get income stream as well, but requires for our project to make new styles and generate tiles. Fortunately, OpenMapTiles project has been working on tiles import and schema, but there are several problems with them which I have been fixing during several months:

* Very long time required to import - 20+ days (24h) of server time
* Problems with POI data (icons were frequently missing and its hard to debug)
* Several other problems as well, some of them I don't even recall now.

Now the import time was a major issue. Import works by first pushing all data from PBF (Planet.PBF) to PostGIS database and later generation of tiles via Mapnik and tilelive-copy. Pushing data into PostGIS takes about 8 hours of intensive CPU use and later ~12+ hours by some threads to reorganize the database. As such, this part is not a problem and has rather reasonable timeframe. Now, when I used OpenMapTiles scripts, there about 20 days of import were estimated.

By default, as usually used, the tilelive-copy (main import command generating tiles) doesn't use all available CPUs for several reasons. Since for OSM Scout Server we need tiles split into smaller databases covering parts of the World, it was simple to make parallel imports and write Makefiles that pushed the server to full usage.

Next, I had to optimize queries, pre-generate as many of them as possible to avoid repetition induced by the same queries run for tiles at different zoom levels, and PostGIS data storage layout.

All these optimizations resulted in dropping tile generation time (after PostGIS import) to about 4*24h, probably a bit less. I expect to get it somewhat faster after realizing that probably 1-2 days were slowed down significantly by huge ice polygons in Alaska, Canada, and Greenland. One way to reduce the polygons would be to run as many imports as possible leading to the increase in overall ambient temperature in these regions, but a faster way would be to split the polygins into the smaller ones. Splitting of the polygons will be tested the next time. My scripts are available at https://github.com/rinigus/mapbox-gl-importer .


In addition to imports, the styles have been reworked. For Mapbox GL, styles are described using a single JSON file which defines all the layers one-by-one. There are nice GUI utilities to work with the style file, but they all suffer from rather large amount of repetitive work. Namely, many layers use the same colors, similar logic, and so on. As a result, there is a lot of point-and-click or emacs/vi replace if you want to make different colorscheme (light and dark), for example.

I have setup a project for the styles used by OSM Scout Server that tries to address some of the repetitive work in describing the styles and modifying them: https://github.com/rinigus/mapbox-gl-styles . The styles are described by collection of layers (https://github.com/rinigus/mapbox-gl...les/src/layers) that are used by all styles, style-specific variables (example for https://github.com/rinigus/mapbox-gl...variables.less) allowing to use LESS color transformations, and the list of layers used by specific style. Thus, for example, style with English names is made by trivial modification of variables file (but that's easy to do with emacs/vi as well by running replace on JSON) and dark style is made by changing colors only.

In future, when Mapbox GL Native will start supporting expressions better, it will be possible to describe style layers with less repetitions than its done now, but that would take some time till Mapbox GL Native will get it implemented.

If you wish to edit styles for OSM Scout Server, let me know. On significant interest, I will try to address the simplicity of visual feedback and providing tiles. At present, I am using an editor, OSM Scout Server on Linux PC, and Mapbox GL QML widget based viewer on Linux. Its surely possible to replace the viewer with web-based one (Mapbox GL JS) and probably make it easier to get tiles for testing.

At present, I included osmbright and osmbright-en as OSM Scout Server styles, with the dark styles expected in the future releases. Plan is to get to parity with Mapnik styles.

Hmm, that turned out to be rather long description of what has been done. Sorry

Enjoy using the server and updated maps.
 

The Following 18 Users Say Thank You to rinigus For This Useful Post: