Notices


Reply
Thread Tools
Posts: 1,414 | Thanked: 7,547 times | Joined on Aug 2016 @ Estonia
#551
Originally Posted by peterleinchen View Post
@rinigus
this is a very special beg (so please feel free to ignore)

I have an exfat formatted SD card (I know, I know ) and (now) around 20G of maps data.
It works quite well with exfat-fuse drivers.

But after having increased the size of datasets from only Germany to Europe I experience long slowdowns at start and also crashes.

It is always for gz files of valhalla.
As I have seen lots of CPU (more than 100%) for harbour-osm-scout-server and mount.exfat I assume timeout of boost::filesystem actions.
If it is really time out would it be possible to extend that for me?
I know boost as a C++ library and found occurences only in mapnik*.so library. So maybe it is possible to check?

And as I fiddled with terminal I got also some other warnings of 'binding loop detected). Please see the screenshot (fingerterm sometimes kind of 'freezes' and I could not copy/paste to text).
I don't have whole Europe on my SD Card, so maybe its some scaling issue. Also, I use ext4 as a FS for SDCard.

Boost is used by Valhalla which is linked into the main program now. From the error messages, I don't see any timeouts, but some interrupted call. Why was it interrupted, I don't know.

Valhalla lib is in major rewrite right now by its devs. As a part of it, they plan to get rid of many dependencies, boost::filesystem included. Maybe that will help.

As for now, I suggest you to look realistically what do you want to use in the next 2 months. I suspect its not full EU. 2 months are from the fact that I am trying to update the maps once in 2 month period.
 

The Following 5 Users Say Thank You to rinigus For This Useful Post:
peterleinchen's Avatar
Posts: 4,117 | Thanked: 8,901 times | Joined on Aug 2010 @ Ruhrgebiet, Germany
#552
okay.

It was just a guess as I felt also slowdowns with bigger file operations. And could not think of any other reason as else exfat is stable.

I was used (times of Nokia maps) to have whole world on device. But at that times is was around 11G. Now with OpenMaps just Germany only is a few GBs. And I like to have a few countries I need to travel to always on device.

But I will (need to learn to) live with it
Anyway, a real big thank you
__________________
SIM-Switcher, automated SIM switching with a Double (Dual) SIM adapter
--
Thank you all for voting me into the Community Council 2014-2016!

Please consider your membership / supporting Maemo e.V. and help to spread this by following/copying this link to your TMO signature:
[MC eV] Maemo Community eV membership application, http://talk.maemo.org/showthread.php?t=94257

editsignature, http://talk.maemo.org/profile.php?do=editsignature
 

The Following 5 Users Say Thank You to peterleinchen For This Useful Post:
Posts: 1,414 | Thanked: 7,547 times | Joined on Aug 2016 @ Estonia
#553
Originally Posted by peterleinchen View Post
okay.

It was just a guess as I felt also slowdowns with bigger file operations. And could not think of any other reason as else exfat is stable.

I was used (times of Nokia maps) to have whole world on device. But at that times is was around 11G. Now with OpenMaps just Germany only is a few GBs. And I like to have a few countries I need to travel to always on device.

But I will (need to learn to) live with it
Anyway, a real big thank you
Well, you could try ext4 as a FS. I don't know how efficient is the caching on Fuse-based FS and I would expect Valhalla to use it properly. Which makes a great use of free RAM, if its available.

When having many datasets, it could also slow down search for POIs and addresses, unless you choose to restrict search to one country / territory only.
 

The Following 4 Users Say Thank You to rinigus For This Useful Post:
pichlo's Avatar
Posts: 6,445 | Thanked: 20,981 times | Joined on Sep 2012 @ UK
#554
Peter, have you tried keeping all maps but switching off using them all at once?
OSM Scout Server -> Settings -> Geocoder-NLP -> Search all available maps
When switched off, it enables a drop-down menu at the top of main OSM Scout Server's screen to select your currently used map.
Not sure if it makes any difference, though. The option's name suggests that it only applies for searching and I am sure rinivus would have mentioned it if he believed it was worth it, but... the map selection in the main screen does not mention any ifs or buts so I would try it
__________________
Русский военный корабль, иди нахуй!
 

The Following 6 Users Say Thank You to pichlo For This Useful Post:
Posts: 1,414 | Thanked: 7,547 times | Joined on Aug 2016 @ Estonia
#555
Originally Posted by pichlo View Post
Peter, have you tried keeping all maps but switching off using them all at once?
OSM Scout Server -> Settings -> Geocoder-NLP -> Search all available maps
When switched off, it enables a drop-down menu at the top of main OSM Scout Server's screen to select your currently used map.
Not sure if it makes any difference, though. The option's name suggests that it only applies for searching and I am sure rinivus would have mentioned it if he believed it was worth it, but... the map selection in the main screen does not mention any ifs or buts so I would try it
It applies to the geocoder only. Other backends (Valhalla and Mapbox ) will be still serving all datasets. Although, I would expect that Valhalla is influenced more than others due to its use of large number of files
 

The Following 5 Users Say Thank You to rinigus For This Useful Post:
peterleinchen's Avatar
Posts: 4,117 | Thanked: 8,901 times | Joined on Aug 2010 @ Ruhrgebiet, Germany
#556
Originally Posted by rinigus View Post
It applies to the geocoder only. Other backends (Valhalla and Mapbox ) will be still serving all datasets. Although, I would expect that Valhalla is influenced more than others due to its use of large number of files
Yep, unfortunately it seems only (or mostly) valhalla is affected.
I tried to set the search to one specific map only. But start time of scout server is a few minutes....
__________________
SIM-Switcher, automated SIM switching with a Double (Dual) SIM adapter
--
Thank you all for voting me into the Community Council 2014-2016!

Please consider your membership / supporting Maemo e.V. and help to spread this by following/copying this link to your TMO signature:
[MC eV] Maemo Community eV membership application, http://talk.maemo.org/showthread.php?t=94257

editsignature, http://talk.maemo.org/profile.php?do=editsignature
 

The Following 5 Users Say Thank You to peterleinchen For This Useful Post:
Posts: 1,414 | Thanked: 7,547 times | Joined on Aug 2016 @ Estonia
#557
In the next few days, there will be an update in OSM Scout Server and its geocoder databases. This update will introduce some changes in the geocoder database format. As a result, it will be impossible to use new server with the old geocoder database and the old server with the new maps.

So, if you need new maps within a week or so, get them now. I will update the server at OpenRepos only after the new format has been pushed to the distribution servers. So, there maybe 12-hours gap when you cannot get new maps.

At present the changes are tested, consider it as an early warning.
 

The Following 11 Users Say Thank You to rinigus For This Useful Post:
Posts: 1,414 | Thanked: 7,547 times | Joined on Aug 2016 @ Estonia
#558
New release is out: 1.12.0

In this release, long waited postal code support was introduced. As a result, geocoder database format has been extended and you have to update the datasets. Press "Check for updates" and proceed with the download. If datasets were up to date, only geocoder-nlp database will be updated.

Postal codes are imported from OSM and, for UK, from https://www.freemaptools.com/. Extra postal codes were reversed geocoded on import to figure out admin hierarchy of them (which part of UK they come from).

When searching with postal codes, you could either specify the code alone or together with some part of hierarchy. When postal code is specified together with address parts, its searching for records that have both - postal codes AND address - components in the record.

Postal codes should be detected by libpostal automatically. For some UK tests that I did, I could see that it was not working. This could be due to older training dataset or its incompleteness from the beginning. Unfortunately, libpostal development has been slow these days, but maybe, together with other users, we can fix few bugs and figure out what to do with the training data.

Primitive parser supports entry of postal codes using prefix "post:" . NB! IT'S CaSe SenSITIVE. So, for example:

post: AA BB CC , city

would be parsed with city as a part of admin and AA BB CC as a postal code. For primitive parser, postal codes can be entered anywhere in the string, just keep that part separated by commas. Documentation at https://rinigus.github.io/osmscout-server/en/ covers this as well.

In addition to postal codes, geocoder now returns phone number and website, if these data were at OSM. Pure Maps will be returning it in the next release, modRana has to be patched. All API is backward compatible, so it works already, just these components are not shown to the users.

I have optimized a bit further loading of libpostal for users of primitive parser only. Would be interesting to see, if @pichlo or others on J1 would get search results faster.

Translations have been updated - great work by all translators, thank you!
 

The Following 10 Users Say Thank You to rinigus For This Useful Post:
Posts: 1,414 | Thanked: 7,547 times | Joined on Aug 2016 @ Estonia
#559
As mentioned yesterday, I published new Valhalla's datasets with public transport data. These are available now as an update of Valhalla's datasets via Map Manager of OSM Scout Server. The server does support it already, so no extra updates of the software is needed. Pure Maps supports public transport routing from the version 1.5.1. To add support for modRana, the costing option has to be added (see Pure Maps code for reference).

The data originates from https://transit.land/. Its a pain to download, took about 2 weeks of running one of Valhalla's programs - all due to slow server on their side. This project is one of the projects by Mapzen and, as far as I understand, is continued by volunteers. Hence the slow data feed, I presume.

Public transit is also not the priority of Valhalla's team at present, according to one of the issues in Valhalla's repo.

So, don't expect miracles, but you are welcome to test. We can still discuss issues over here and file them under Valhalla's repos. Its my first import with these data and its possible that something went wrong while downloading. So, some of the issues could be from the data import too.

From testing in Tallinn, it looks to be using busses only and ignoring other means of transport. But maybe its better in other places.

Enjoy!
 

The Following 10 Users Say Thank You to rinigus For This Useful Post:
Posts: 56 | Thanked: 146 times | Joined on Sep 2013 @ Norway
#560
Testing navigating with public transport here i Oslo. It always results i logg error: Exception in Valhalla: Cannot reach destination - too far from a transit stop. Just for you information. I don't expect you to prioritize this. It seems you work very hard too solve other issues.
 

The Following 6 Users Say Thank You to PamNor For This Useful Post:
Reply

Tags
geocoder, linux, offline maps, router, sailfish os, tiles

Thread Tools

 
Forum Jump


All times are GMT. The time now is 21:52.