Demoserver - call for hosting volunteers

Question

Demoserver - call for hosting volunteers

danpat opened this issue 5 years ago · 19 comments

On Mar 09, 2020, the TLS certificate for router.project-osrm.org is going to expire. At the same time, Mapbox is going to shutdown our hosting of that service. As many of you have probably noticed, our investment in the OSRM codebase has greatly decreased over the last year.

This is a call to the community to see if someone else is willing to take up the mantle for running the demo server. If nobody steps up, then the service will stop working on Mar 09, 2020.

Below is a chart that shows the request load for the last two weeks - the value here is "requests per minute".

I have also posted this notice on the osrm-talk mailing list for wider discussion.

Answer 1 · 2020-01-28T16:43:01.000Z

This might be worth cross-posting to a wider audience, such as the main talk list. I also posted a mention in OSMUS Slack.

Note that router.project-osrm.org is distinct from router.openstreetmap.de.

Answer 2 · 2020-02-05T16:15:23.000Z

I can host the demo server.

Here is an existing one - Germany Car profile:

I have also posted on the OSRM mailing list (https://lists.openstreetmap.org/pipermail/osrm-talk/2020-February/001837.html)

Answer 3 · 2020-02-10T12:54:46.000Z

What are the current host specs?

Answer 4 · 2020-02-11T17:00:07.000Z

What are the current host specs?

If you are asking me, I'm using a 4GB RAM with a 2 vCPU server. I am hosting Germany's Car profile. Since there is very little traffic, I didn't need higher specs for now.

Answer 5 · 2020-02-12T20:07:39.000Z

No, i asked @danpat for server specs, so i'll be able to estimate the expenses.

Answer 6 · 2020-02-13T03:24:42.000Z

Unfortunately, I don't have a great answer to the size question. Currently, the demoserver is running on Mapbox's shared infrastructure, and we've partitioned the data out across several machines (smaller geographic regions), which also serve some of our other traffic, so I can't easily point to one box in our system and say "that's the demoserver". We've simply absorbed it as overhead.

If I were contemplating running the service globally, I'd be aiming for:

Enough RAM to load all files into memory (about 70GB for a car.lua CH dataset last time I looked, but I haven't processed the whole planet for a while)
Enough CPU cores to keep up with the demoserver request volume (250/sec). Estimating 50ms per request, you'd need 12-16 cores to do that.

Here's the daily network traffic for the last couple of weeks to get an idea of bandwidth volumes. The p?? values are response time percentiles, in seconds. This is across all request types (/route, /table, /match, /trips, /nearest):

	dt	total_requests	outgoing_bytes	incoming_bytes	p50	p90	p95	p99
1	2020-02-12	20957484	32020135708	8899911485	0.007	0.052	0.052	0.053
2	2020-02-11	18375314	34171759908	7888859078	0.007	0.052	0.052	0.054
3	2020-02-10	17536583	37150229241	7326692479	0.007	0.052	0.052	0.054
4	2020-02-09	11766243	31729906622	4949341021	0.007	0.051	0.052	0.053
5	2020-02-08	12630320	33114101719	5324081819	0.007	0.052	0.052	0.054
6	2020-02-07	18440429	41877401205	7786411052	0.008	0.052	0.052	0.054
7	2020-02-06	16306322	45444040204	6970381051	0.007	0.052	0.052	0.054
8	2020-02-05	15910468	40002820959	6950761916	0.007	0.052	0.052	0.054
9	2020-02-04	16412076	35710951466	6948442645	0.007	0.052	0.052	0.054
10	2020-02-03	15240168	43539650561	6408017674	0.007	0.052	0.052	0.053
11	2020-02-02	11824870	38352572280	5136492785	0.007	0.051	0.052	0.054
12	2020-02-01	13011131	41814895623	5702334096	0.007	0.052	0.052	0.054

You also need to factor in the data processing costs - the expense here depends highly on how often you refresh the map. Currently it's done daily. You'd want 256 or more GB of RAM for the fastest osrm-extract/osrm-contract turnaround time, and the more CPUs you can give it the better. I think you can turnaround a planet-sized CH dataset in 12-16 hours with a 32 core machine and 256GB of RAM.

Answer 7 · 2020-02-14T13:25:26.000Z

@danpat It looks more demanding than i expect. I sent you a mail with more private details.

Answer 8 · 2020-02-15T15:25:57.000Z

It's too much for me. Sorry, I had to drop my proposal.

Answer 9 · 2020-03-02T08:56:27.000Z

@danpat We still can do that, but i can't reach you to discuss the details

Answer 10 · 2020-03-12T13:07:03.000Z

Hello,
What is the current situation? What is the foreseen future of the demo server?
As of today it seems that the http://router.project-osrm.org/ server is still running.

Answer 11 · 2020-03-12T15:05:17.000Z

Sorry, I've been totally slammed by life and work.

Yes, the demoserver is still running, it's not being turned off on our side (yet, no direct timeline for that yet, but it'll happen eventually), but the https:// certificate (if you access it over https) has expired and won't be renewed by Mapbox.

A couple of people have offered to host a box to replace it, I need to find time to follow-up with them.

Answer 12 · 2020-03-21T13:26:42.000Z

@danpat I opened the following pull request to fix the demo site for now: Project-OSRM/osrm-frontend#319

Answer 13 · 2020-03-21T15:24:19.000Z

@danpat A suggestion: How about splitting this country-wise (or region wise) and restructuring on the demo website also to let the user pick a country first? Then things can be divided among volunteers. Or you can just add links on the site to take users to country-specific URLs.

I'd like to volunteer to host OSRM for India. I'm using OSRM (our own hosted) heavily in my work and have even customized the profile a bit to have left-side driving and to permit pedestrians on main roads. I have a digitalocean server droplet mapped to my personal domain and have free LetsEncrypt SSL on it . It's working fine since over 2 years. Here's a page I'm hosting that has a backend daily fetching latest OSM data and processing it: https://server.nikhilvj.co.in/OSM/roads.html

Additional request : Permit the country-wise volunteers to put a small link to accept contributions towards the server or contact the hoster for professional enquiries. This way if there are heavier users in a particular place, they can get in touch with the hoster, take ownership instead of socializing the burden to the world. (I have a hunch that 80% of your usage load is coming from very few people who have hooked it up to their forever-running programs - if we split by region, majority of places probably won't have much traffic)

I know this isn't a fancy full-planet solution, but hey if you have to choose between complete shutdown and decentralizing, why not decentralize.

Answer 14 · 2020-03-25T12:12:32.000Z

@answerquest That is a good idea. But it requires a bit of development on the demo site. Would anyone be willing to do this? I incorporated a drop down for routing.osm.ch and routing.openstreetmap.de but that was already time consuming and too much javascript for me.

Answer 15 · 2020-05-06T06:48:56.000Z

Can we not use https://routing.openstreetmap.de?

Answer 16 · 2020-05-31T14:10:50.000Z

map.project-osrm.org is now switched to the fossgis server as the backend. https://github.com/fossgis-routing-server/osrm-frontend

Answer 17 · 2020-06-04T23:33:50.000Z

Looks like somebody redirected router.project-osrm.org at router.openstreetmap.de as well.

Answer 18 · 2020-06-04T23:50:37.000Z

Looks like somebody redirected router.project-osrm.org at router.openstreetmap.de as well.

It was announced few comments above I suppose?

Answer 19 · 2020-06-05T00:06:39.000Z

@mandeepsandhu The comment only mentioned map.project-osrm.org, not the router.project-osrm.org CNAME (first one hosted the osrm-frontend project via Github Pages, the second one hosted an osrm-routed API instance).