Ok, I've got a project update. At first I had been all giddy about this, then trashed the idea after struggling fruitlessly with GDBM. But after Gnuite told me about a future DB change that'd once again make this script possible, I began dev work on it again. And since I believe in the principles of FOSS, I'm going to share the source code as I work on it to get some input, help and to improve it greatly.

To start with I came up with a simple engine for converting a decimal number to a grid address ID. The principle behind this is that you'll start with a zoom level, say level 16 (I'm counting zooms based on the way the grid addressing works with level 20 being a full 4 quadrant map of the earth) and going down to level 4 (you don't need any zooms higher than level 4). That's 12 levels. That's also one bloody heck of a lot of files. I did a test with 200 files and it came out to about 1mb average size for the entire lot.

It'll also pull 5460 tiles for the first six levels you go down. It takes close to 1 million tiles or about 5gb to do the entire state of Michigan. I could be off by a little bit, but that's a pretty fair estimate. And that's starting with just one grid tile on one zoom level as your focal point for starting the script. So the total file size and number of tiles is significantly smaller than I had first estimated. Mostly because the math works out a lot differently.

But anyways, here's the address engine that I have right now. Again, it's a bit rough, but this is a first effort and I figure that if we work together long enough this will evolve into a full fledged downloader script. Now, given everything I've learned so far, you're looking at having to run this script for close to 6 hours to do just a state level snapshot. But that certainly beats the 18-32 hours it would take with the Nokia tablet.

Also, given that maemo mapper will eventually be going to an sqllite database, I figure we may as well start building the downloader to import to that format of database. The only three areas I can really think we need to work on right now (I'm not saying they're the only areas, just the most immediate three) is the downloading module for grabbing the files, the sqllite database handler and importer, and an initial config section.

The config section would hold all the user editable configs for the script obviously, as most of these configs can't really be passed from the command line. Now it's also reasonable to think that these could be put out in a separate file and handled that way so as to avoid requiring end users to edit the actual parent script. So, anywho, without further adieu, here's the address engine. Now mind you there's some testing stuff in here still, so feel free to ignore that.

Ok, now with that out of the way, who's interested in helping me develop this. I'm sure gnuite would appreciate it if we could do the dev work instead and take that load off his hands.

edit: Ok, minor correction. It looks like I was estimating too small again. It looks like my original data and tile numbers are gonna be much closer to the mark.

Lord Raiden, you have no idea!

At Google's finest detail level, the mercator projection divides the earth into a grid that is 2^19 tiles wide and 2^19 tiles tall. That's 274 billion tiles for just one zoom level!

Now, Google doesn't map the entire world at that zoom level, and a lot of it is just water, so it's not quite that simple, but you get the idea.

This is, after all, what made the idea so revolutionary when Google Maps first came out. Up until then, all the maps were only generated upon request, which was slow. Generate all the maps up-front, however, and all you have to do is HTTP GET the images. It was a brilliant idea, but it obviously requires a lot of disk space.

Yeah, you ain't kidding it is. It's almost making me wonder if it wouldn't be more practical to go every other zoom level to reduce the total number of tiles needed to be downloaded. I redid my numbers again, taking into account for the funky way the addresses add up and came out to something like 49 million total tiles rather than the 2.8 I first came up with.

Since that's WAY too frigging many tiles for one person to download, I've considered making this a community project with numerous people downloading tiles, building databases, and then sending those in to be merged into a master database. Again, it's just an idea and would have to be updated yearly to adapt to changing roads and area maps, but it'd be progress.

That's what I do for a pretty good-sized area. Only regions of of particular interest get every level; but it doesn't help all that much.

I don't need nearly that much. I generally download only one zoom level, maybe 2. One level for normal tracking, and maybe one big enough to see the whole route at one time. I see no need to have all zoom levels, or even half, for an entire area, or even one state.

I do have the entire US, all zoom levels, on one 1GB SD card, using vector maps, though. There are reasons vector maps are much better - size and autorouting capability being two. They aren't as pretty, though, nor as cheap. I don't bother with the basic VE maps, just the hybrids, so I can view the photos. For actual driving directions, I have a vector-based program on a Palm and a standalone GPS. MM on a Nokia is just for fun, and I would never tie up terabytes of storage for maps for it.

Wow, now if there were a way to get ahold of a full data dump of vector maps for the US, that'd rock. The jpg/png thing is pretty for certain, but with that much data and size, someone's gotta come up with something different.

That is of course, unless gnuite, or one of the other devs here knows of a way to do compression in a database. Sure, that's still a lot of files to download, but spread across a large group, and done one time, or just once a year, and with enough compression, it's feasible enough to think that we could do it with the existing image map system. Otherwise if the answer is no, then it may be prudent to consider the vector maps you suggest.

BTW, not to sound dumb or anything, but why are the vector maps a lower quality if they're dynamic like I suspect they are?

The maps I have are from Mapopolis. They're fine for maps, but you can't get photos or anything like that. TomTom and other companies sell vector maps, and most, if not all, total <1GB. The format is different from raster maps, which are just graphic images. You can get vector maps of the US from the Census Bureau, and this is what the free mapping programs which provide on-the-fly routing use. They're missing some data, though, such as one-way streets, and you can easily be routed on the wrong side of an interstate highway, or the wrong way on a ramp or a one-way street. Getting all that data costs time and money, and that's why you have to pay for a robust mapping program. You're paying for the data, mostly. Mapopolis provided the actual mapping program for free, charging just for the maps. Sadly, they''re no longer in business. PDA mapping programs are quickly becoming extinct, because you can buy a standalone unit cheaper than you can get the maps alone.