Proposal: Explore API

[bboure]

Hi there,

I would like to suggest an /explore api. It would be similar to the /reverse and /nearby apis, but differ from them by taking a bbox instead of a point.

Use-case

When browsing a map, this api would return the features that are included in the currently visible area.

Proposal

Required parameters:

• boundary.rect.min_lat
• boundary.rect.max_lat
• boundary.rect.min_lon
• boundary.rect.max_lon

Available filters: same as /nearby and /reverse

Features could be sorted by popularity (when available), or alternatively, by distance from the center.

Comments/suggestions are welcome.

[missinglink]

Hi @bboure, thanks for the suggestion.

It would be simpler to add the bbox filter to /reverse and /nearby than create a new API.

The problem with making a new API is that we need to build up a range of tests and document it etc.

Do you see any pros/cons to having a new API vs extending the existing ones to cover this use case?

[bboure]

Thanks for your feedback @missinglink

I don't see cons in extending /reverse and /nearby. I also thought about it, but then I thought that it was a slightly different concept and that the api names could become misleading.

If we do that. I guess that the point parameter should become optional on both endpoints, and add a rect parameter. One of the 2 should be required, and possibly mutually exclusives.

The rest should not really change I guess.

If that is something you would see useful, I would be happy to try and open a PR.

[missinglink]

Yeah it sounds like a reasonable change to those APIs and it would solve your use-case, so yeah 😄

A couple of questions about your proposal:

1. Changing the point param to optional would be a breaking change to the API and so might cause some friction. If we could leave this unchanged it would be much easier to merge.

2. In the case of a very large bbox (like zoomed out to encompass all of Europe) there would presumably be a maximum of a few hundred results returned for performance reasons?

If the point param is provided then these would be reduced to the ~10 closest points to the target.

You mentioned above the popularity score, but right now this is not very widely used in Pelias, generally only for airports.

So I suspect (without seeing it in action) that what you are proposing would show the top 10 airports in Europe, which probably isn't what you wanted to see?

In the case of a bbox on Europe, what would be your ideal top 10 results?

[Joxit]

Hi there,

I think this use-case is what nearby should do right ?
As I can see, neraby do the same job as reverse without pip-service.
This endpoint is still in "alpha" and is not documented... This will give us more possibility for breaking changes than reverse.

IMO, we should add this new feature only in nearby and not in reverse 🤔 (and start document it 😅).

We should probably also add categories with this :) (all shops in this bbox, bus stations in this bbox...)

[bboure]

@missinglink

Changing the point param to optional would be a breaking change to the API and so might cause some friction. If we could leave this unchanged it would be much easier to merge.

Hum, I guess that point could be set to the center of the bbox, but that would be annoying for consumers to be forced to specify it.

Conceptually, I also see a difference between: "give me all features in the bbox" and "give me the features that are near a point limited to the given bbox".

Why do you think this would be a breaking change? I understand the opposite would be the case.

In the case of a very large bbox (like zoomed out to encompass all of Europe) there would presumably be a maximum of a few hundred results returned for performance reasons?

Yes, there should be at least a default limit (TBD. e.g: 100), and possibly a hard limit to avoid performance issues.

You mentioned above the popularity score, but right now this is not very widely used in Pelias, generally only for airports.

My ideal top 10 results would be most famous places in Europe. For example:

• Eiffel tower
• Big Ben
• Colosseum
• Le Louvre
• The Acropolis

etc.

This is why I suggested the popularity feature.
I know that this is not a feature that is currently well implemented.

A possible alternative for this for now would be to show them by nearest place from the center. But here, we would end up other issues.
When zooming out on europe, top 10 features would be concentrated in the middle of the area. That would not be expected either.

On top of my head, I don't see any other alternative.

[bboure]

Thanks @Joxit for your feedback.

We should probably also add categories with this :) (all shops in this bbox, bus stations in this bbox...)

Yeah, this is in fact exactly what I would like to achieve, in the end.

[missinglink]

Why do you think this would be a breaking change? I understand the opposite would be the case.

It's true, I can see both sides of that 😉

A purist might go by the classic definition of reverse as taking only points as input:

"Reverse geocoding is the process of back (reverse) coding of a point location (latitude, longitude) to a readable address or place name."

Sorry I'm totally thinking out loud here as this is still early stages and its easy to get stuff right now in words rather than change the code later 😁

One thing I'm a bit worried about with having this in /reverse is potentially very different performance characteristics to the query depending on the input params which would make rate limiting harder for enterprise users

Anyhoo, I'm not trying to discourage anything, I'm just saying that when it comes time to merge changes to /v1/reverse these are some of the things that might come up.

On the other hand the nearby endpoint sounds more like what you're after, and isn't really officially supported so it'll be much easier to make the changes you want 😉

How about we start there and once that's done we take another look at reverse?

[bboure]

How about we start there and once that's done we take another look at reverse?

Sounds good.

I am willing to try and open a PR about this soon.

Before anything it would be great to clarify a few things on paper.

As I understand, starting from here

• Add a rect parameter, taking min/max lon/lat.

• point becomes optional on its own but either point or rect should be provided.

• Sorting: It looks like you guys are having a look back at venue popularity openstreetmap#493
  As I already mentioned, it sounds like popularity could be a good candidate when fully implemented and rect is provided.

Side note: Popularity could also become useful with point in order to give more visibility to venues that might be a little farther but more relevant/popular.
e.g.: I am looking for a restaurant. I might prefer that good restaurant 500m away rather than the small bar that serves frozen pizza right next to me 🙂

This could come as an extra parameter. Maybe something like boost: distance or popularity.

• size parameter, should be limited to 10 by default (already the case) to avoid too many results on big areas.

• categories/layers should keep working too.

• reverse should remain unchanged.

Also, I am still getting familiar with the codebase and I noticed that nearby uses the same query as reverse under the hood. Is there anything to take into consideration in order to not break reverse?

[missinglink]

Yep we're looking at the popularity scoring issue again, it's become a priority for a client so I should hopefully get some time to clean that up and merge it. I think that'll make this feature much more powerful 🚀

I'm still a little uneasy about the "point becomes optional on its own but either point or rect should be provided" API as I find this sort of thing confusing to use (as an end user).

Where possible we should aim for a simple API where params are either required or optional but never required unless xx then optional, this leads to confusion and can also make the code and tests confusing and sometimes messy to write and maintain. Any client-side libraries would also need to be aware of this logic to correctly format requests against the API.

But also I see your use-case, like when you're zoomed out you'd want to see the results ranked by importance rather than proximity.

One simple option is to set the point.lat=0 and point.lon=0. Any points > 100km from 0,0 (Null Island ™️) will not receive any proximity boost and the effect is what you're looking for, this is functionally equivalent to omitting the point.* params.

It's not very elegant but it works as-is without any dev work.
You could add boundary.rect (and make it optional) to further filter the results to the available screen space.

[missinglink]

One thing to be aware of there is https://github.com/pelias/api/blob/master/sanitizer/nearby.js#L15, this line uses the same point sanitizer as reverse, which generates an artificial boundary.circle around the point when one has not been specified, I dont recall the exact reason but that's likely there for performance reasons.

[missinglink]

https://pelias.github.io/compare/#/v1/reverse?point.lat=40.74&point.lon=74&debug=1

[bboure]

One thing to be aware of there is pelias/api:sanitizer/nearby.js@master#L15 , this line uses the same point sanitizer as reverse, which generates an artificial boundary.circle around the point when one has not been specified, I dont recall the exact reason but that's likely there for performance reasons.

Not sure what you meant here. What I understand from the code is that is translates the point parameter into boundary.point which is what the query expects.

One simple option is to set the point.lat=0 and point.lon=0
Any points > 100km from 0,0 (Null Island ™️) will not receive any proximity boost

That would be ok with me and still work.
Is the 100km limit hard-coded somewhere? Default radius filter seems to be 1km
You last screenshot seems to show that too.

Also, wouldn't the radius filter out results if the rect is outside the radius?

[missinglink]

I'm a little rusty on this myself so forgive me as I remember how it's configured.

If you have a look in debug mode at a query such as /v1/reverse?size=40&point.lat=0&point.lon=0&debug=1
https://pelias.github.io/compare/#/v1/reverse?size=40&point.lat=0&point.lon=0&debug=1

.. then you'll see there are two query sections, the one I screenshotted above to do the filtering:

and this one for the sorting:

The query clause in the top screenshot is generated in https://github.com/pelias/api/blob/master/sanitizer/_geo_reverse.js#L24-L35 and could be skipped in the case where a boundary.rect was provided instead.

Regarding the 100km, I seem to be wrong about that as I don't see that in the debug output. The total distance is controlled by "distance": "1km" and I think that value is quite low for performance reasons in densely popularity places like NYC.

The current query is very fast as its able to produce a small list of 'hits' that match the filter condition and then only has to sort those hits, this is crucial for performance at scale.

What you're proposing is to (at some scales, like zoomed out to see the whole world) have every record matched in the 'hits' (which can be >500,000,000) and then sort each of those records by popularity. This query will most likely be very slow due to the amount of disk I/O (all the popularity info has to be loaded from disk into RAM) and also a lot of CPU to sort the whole index by popularity, IIRC elasticsearch isn't able to cache these sorting conditions.

[missinglink]

Try this elasticsearch query on a fairly large Pelias index (with a few hundred million records) and see what the performance is like, I could be wrong 🤷

POST http://localhost:9200/pelias/_search

{
  "query": {
    "bool": {
      "filter": [
        {
          "geo_bounding_box": {
            "center_point": {
              "top_left": {
                "lat": 89,
                "lon": -179
              },
              "bottom_right": {
                "lat": -89,
                "lon": 179
              }
            }
          }
        }
      ]
    }
  },
  "sort": [
    "popularity"
  ]
}

Performance will likely change once there are more records with a popularity, at the moment there are only ~1000 airports with a popularity set.