Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Maps] GeoJSON upload can't index polygons #61794

Closed
jsanz opened this issue Mar 30, 2020 · 18 comments
Closed

[Maps] GeoJSON upload can't index polygons #61794

jsanz opened this issue Mar 30, 2020 · 18 comments
Assignees
@kindsun
Labels
bug Fixes for quality problems that affect the customer experience [Deprecated-Use Team:Presentation]Team:Geo Former Team Label for Geo Team. Now use Team:Presentation

Comments

@jsanz
Copy link
Member

jsanz commented Mar 30, 2020

Kibana version:
7.6.1, master

Elasticsearch version:
7.6.1, snapshot

Server OS version:
Ubuntu 9.10

Browser version:
Version 1.5.115 Chromium: 80.0.3987.149 (Official Build) (64-bit)

Browser OS version:
Ubuntu 9.10

Original install method (e.g. download page, yum, from source, etc.):
Docker image, from source

Describe the bug:
A polygon dataset is not being successfully indexed by GeoJSON Upload.

Steps to reproduce:

  1. Download Leeds wards GeoJSON file from this URL linked on this open data portal
  2. Index it using GeoJSON Upload tool.
  3. The last step of the wizard states that the index is created but the button to add the layer is not enabled and the generated index has no documents.

Expected behavior:

The documents are created and layer can be added to the map.

Screenshots (if relevant):

image

Errors in browser console (if relevant):
I can't see any errors on the browser

Provide logs and/or server output (if relevant):
I can't see any errors on the Kibana or ES logs when running from source, on the docker image I could see these errors

trace 
testing-elasticsearch-761 | {"type": "server", "timestamp": "2020-03-30T12:09:01,604Z", "level": "INFO", "component": "o.e.c.m.MetaDataCreateIndexService", "cluster.name": "testing-elasticsearch", "node.name": "master.elasticsearch", "message": "[leeds_wards_4] creating index, cause [api], templates [], shards [1]/[1], mappings [_doc]", "cluster.uuid": "qaWPq407TVaXX-EXhhP1rw", "node.id": "zTHeleceQzSfrJ7X8CGfhw"  }
testing-elasticsearch-761 | {"type": "server", "timestamp": "2020-03-30T12:09:01,789Z", "level": "DEBUG", "component": "o.e.a.b.TransportShardBulkAction", "cluster.name": "testing-elasticsearch", "node.name": "master.elasticsearch", "message": "[leeds_wards_4][0] failed to execute bulk item (index) index {[leeds_wards_4][_doc][ne5YK3EBQ4SwjTZbVspM], source[n/a, actual length: [30.3kb], max length: 2kb]}", "cluster.uuid": "qaWPq407TVaXX-EXhhP1rw", "node.id": "zTHeleceQzSfrJ7X8CGfhw" , 
testing-elasticsearch-761 | "stacktrace": ["org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [coordinates] of type [geo_shape]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.AbstractGeometryFieldMapper.parse(AbstractGeometryFieldMapper.java:446) ~[elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:488) ~[elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:505) ~[elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:418) ~[elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:395) ~[elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:112) ~[elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:71) ~[elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:267) ~[elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:793) ~[elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:770) ~[elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:742) ~[elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:267) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:157) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:202) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:114) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:81) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:895) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:109) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:374) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:297) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$24(IndexShard.java:2791) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:113) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:285) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:237) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2765) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:836) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:293) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:256) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:306) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:344) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at java.util.ArrayList.forEach(ArrayList.java:1507) [?:?]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:19) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at java.util.ArrayList.forEach(ArrayList.java:1507) [?:?]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:71) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at java.util.ArrayList.forEach(ArrayList.java:1507) [?:?]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:82) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at java.util.ArrayList.forEach(ArrayList.java:1507) [?:?]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:23) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at java.util.ArrayList.forEach(ArrayList.java:1507) [?:?]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:102) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:345) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:322) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:263) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:616) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:591) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.RBACEngine.buildIndicesAccessControl(RBACEngine.java:507) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:298) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:653) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.AuthorizationService.resolveIndexNames(AuthorizationService.java:554) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:251) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:653) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:329) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:247) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:651) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:250) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:651) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:290) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:314) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:287) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:261) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:227) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:193) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:117) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:244) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:123) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:111) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:195) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:130) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:248) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:310) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:321) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:245) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:196) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:139) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:121) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:313) [x-pack-security-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:750) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
testing-elasticsearch-761 | "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
testing-elasticsearch-761 | "at java.lang.Thread.run(Thread.java:830) [?:?]",
testing-elasticsearch-761 | "Caused by: org.locationtech.spatial4j.exception.InvalidShapeException: Provided shape has duplicate consecutive coordinates at: (POINT (-1.37699 53.67357))",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.GeoShapeIndexer.concat(GeoShapeIndexer.java:694) ~[elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.GeoShapeIndexer.ring(GeoShapeIndexer.java:507) ~[elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.GeoShapeIndexer.createEdges(GeoShapeIndexer.java:458) ~[elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.GeoShapeIndexer.decompose(GeoShapeIndexer.java:357) ~[elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.GeoShapeIndexer.access$200(GeoShapeIndexer.java:59) ~[elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.GeoShapeIndexer$1.visit(GeoShapeIndexer.java:172) ~[elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.GeoShapeIndexer$1.visit(GeoShapeIndexer.java:79) ~[elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.geometry.Polygon.visit(Polygon.java:96) ~[elasticsearch-geo-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.GeoShapeIndexer.prepareForIndexing(GeoShapeIndexer.java:79) ~[elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.GeoShapeIndexer.prepareForIndexing(GeoShapeIndexer.java:59) ~[elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.AbstractGeometryFieldMapper.parse(AbstractGeometryFieldMapper.java:434) ~[elasticsearch-7.6.1.jar:7.6.1]",
testing-elasticsearch-761 | "... 160 more"]

Any additional context:
Reported by @kevinkraft at https://discuss.elastic.co/t/geojson-upload-hangs-on-creating-index/225661
@jsanz jsanz added bug Fixes for quality problems that affect the customer experience [Deprecated-Use Team:Presentation]Team:Geo Former Team Label for Geo Team. Now use Team:Presentation labels Mar 30, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-gis (Team:Geo)

@jsanz jsanz mentioned this issue Mar 30, 2020
Closed
@thomasneirynck thomasneirynck mentioned this issue Mar 30, 2020
Closed
@thomasneirynck
Copy link
Contributor

This seems to be an underlying Elasticsearch issue. elastic/elasticsearch#54441

@jsanz
Copy link
Member Author

jsanz commented Mar 30, 2020

@kevinkraft as a workaround I managed to index the wards geojson file after passing it through mapshaper.org: just import and then export it again and it should be OK in Kibana.

@kevinkraft
Copy link

kevinkraft commented Mar 30, 2020 •
edited
Loading

@jsanz It isn't working yet. How long should the "Write to index" take?

What options did you use on the mapshaper site?

@jsanz
Copy link
Member Author

jsanz commented Mar 31, 2020

For the Leeds file, on my personal laptop, it indexes it immediately.

I did a quick (1 minute) video of the processing using mapshaper cli, I hope it helps.

Those geojson files follow a old spec that includes a csv section, but also repeated coordinates on those features; we are pretty sure that the latter is the root cause for failing when being parsed by elasticsearch, please subscribe to elastic/elasticsearch#54441 if you want to track the discussion there.

In fact, if you run the mapshaper command with the -verbose option you can see all the things are wrong in that geojson file:

$ mapshaper -i leeds_wards.geojson -clean -verbose -o leeds_wards.fix.geojson
[i] Importing: leeds_wards.geojson
[i] Warning: reversing a CCW ring
[i] Warning: reversing a CCW ring
[i] Warning: reversing a CCW ring
[i] Warning: reversing a CCW ring
[i] Warning: reversing a CCW ring
[i] Warning: reversing a CCW ring
[i] Warning: reversing a CCW ring
[i] Warning: reversing a CCW ring
[i] Warning: reversing a CCW ring
[i] Warning: reversing a CCW ring
[i] Warning: reversing a CCW ring
[i] Warning: reversing a CCW ring
[i] Warning: reversing a CCW ring
[i] Warning: reversing a CCW ring
[i] Warning: reversing a CCW ring
[i] Warning: reversing a CCW ring
[i] Warning: reversing a CCW ring
[i] Warning: reversing a CCW ring
[i] Warning: reversing a CCW ring
[i] Warning: reversing a CCW ring
[i] Warning: reversing a CCW ring
[i] Removed 117 duplicate points
[i] - 56ms
[clean] Find mosaic rings 5ms
[clean] Detect holes (holes: 0, enclosures: 1) 3ms
[clean] Retained 21 of 21 features
[clean] - 63ms
[o] Wrote leeds_wards.fix.geojson
[o] - 23ms

@kevinkraft
Copy link

@jsanz That's great thanks. It worked for the Wakefiled geojson, but it didn't work for the Leeds geojson. I used an alternate file from another website (https://martinjc.github.io/UK-GeoJSON/), and used the map shaper CLI, and it worked.

@jsanz
Copy link
Member Author

jsanz commented Apr 1, 2020

Glad you get it working, thanks for reporting; this will help us to make GeoJSON upload more resilient to somehow invalid files.

@michaelimfeld
Copy link

Hi

I'm having the same issue, my input data comes from: TopoJSON.

I did the following:

  1. Built build/ch-municipalities.json TopoJSON file
  2. Converted it to GeoJSON using https://mapshaper.org/
  3. Ran mapshaper cli:
root@f0ee8c17ea9f:maps# mapshaper -i municipalities.json -clean -verbose -o ch-municipalities.fixed.json
[i] [i] Importing: municipalities.json
[i] [i] - 229ms
[clean] [clean] Find mosaic rings 37ms
[clean] [clean] Detect holes (holes: 8, enclosures: 1) 38ms
[clean] Retained 2,369 of 2,369 features
[clean] [clean] - 419ms
[o] Wrote ch-municipalities.fixed.json
[o] [o] - 152ms

After uploading it to Kibana it seems to hang on "Writing to index: ch-municipalities".
My final municipalities.fixed.json can be found here.

Any ideas?

@jsanz
Copy link
Member Author

jsanz commented Apr 21, 2020

@michaelimfeld your geojson file is not in WGS84 coordinate system, you need to follow the repo instructions on having the coordinates in lat/lon using the REPROJECT=true parameter when running make.

In any case, we definitely can do better on reporting issues in the upload interface.

@michaelimfeld
Copy link

Thanks for the quick reply @jsanz. RTFM :P
Unfortunately I get the same behaviour with the fixed GeoJSON.

Is there any way to debug this? Neither the Elasticsearch nor the Kibana container outputs any error messages.

@jsanz
Copy link
Member Author

jsanz commented Apr 22, 2020

@michaelimfeld I got no problems on importing that file in 7.6.2, on any offending file you'd see errors in the Elasticsearch log. On your original file, the last lines of the ES log were like this:

log 
testing-elasticsearch-761 | {"type": "server", "timestamp": "2020-04-22T09:05:31,003Z", "level": "DEBUG", "component": "o.e.a.b.TransportShardBulkAction", "cluster.name": "testing-elasticsearch", "node.name": "master.elasticsearch", "message": "[ch-municipalities-test][0] failed to execute bulk item (index) index {[ch-municipalities-test][_doc][zKUioXEBhhvZ0_hagc-C], source[{\"coordinates\":{\"type\":\"polygon\",\"coordinates\":[[[555405,138265],[555340.2925000004,138419.0562499985],[552795.5562499985,141691.36250000075],[552529,141944],[551290,138318],[551288,138307],[551279,138276],[551385,138265],[551445,138205],[551625,138075],[551816,137972],[551845,137955],[551985,137915],[552123,137905],[552185,137875],[552305,137765],[552585,137645],[552685,137545],[552766,137495],[552945,137465],[553065,137435],[553105,137415],[553305,137405],[553465,137445],[553525,137445],[553905,137545],[554085,137575],[554125,137595],[554285,137637],[554525,137637],[554586,137626],[554645,137615],[554865,137506],[555106,137348],[555145,137314],[555205,137285],[555325,137325],[555395,137475],[555395,137495],[555455,137615],[555455,137695],[555435,137755],[555435,137794],[555416,137835],[555375,138013],[555315,138173],[555315,138235],[555325,138255],[555405,138265]]]},\"name\":\"Lac Léman (VS)\",\"id\":9759}]}", "cluster.uuid": "qaWPq407TVaXX-EXhhP1rw", "node.id": "zTHeleceQzSfrJ7X8CGfhw" , 
testing-elasticsearch-761 | "stacktrace": ["org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [coordinates] of type [geo_shape]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.AbstractGeometryFieldMapper.parse(AbstractGeometryFieldMapper.java:446) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:488) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:505) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:418) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:395) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:112) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:71) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:267) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:793) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:770) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:742) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:267) [elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:157) [elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
testing-elasticsearch-761 | "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
testing-elasticsearch-761 | "at java.lang.Thread.run(Thread.java:830) [?:?]",
testing-elasticsearch-761 | "Caused by: java.lang.IllegalArgumentException: Unable to Tessellate shape [[0.9437500014901161, -139.70749999955297] [-31.362500000745058, -164.44375000149012] [76.0, -71.0] [78.0, 130.0] [67.0, 128.0] [36.0, 119.0] [25.0, -135.0] [-35.0, -75.0] [-15.0, 105.0] [88.0, -64.0] [75.0, -35.0] [35.0, 105.0] [25.0, -117.0] [-5.0, -55.0] [-65.0, 65.0] [55.0, -15.0] [25.0, 85.0] [-25.0, 166.0] [-55.0, -15.0] [-85.0, 105.0] [-75.0, 145.0] [-65.0, -15.0] [-75.0, 145.0] [-75.0, -155.0] [25.0, -135.0] [55.0, 45.0] [75.0, 85.0] [63.0, -115.0] [63.0, 125.0] [74.0, -174.0] [85.0, -115.0] [-14.0, 105.0] [-8.0, -14.0] [26.0, 25.0] [55.0, 85.0] [15.0, -155.0] [-45.0, -85.0] [-25.0, -85.0] [85.0, -25.0] [5.0, -25.0] [-55.0, -45.0] [-86.0, -45.0] [-45.0, -64.0] [47.0, -105.0] [-67.0, -165.0] [-5.0, -165.0] [15.0, -155.0] [25.0, -75.0] [0.9437500014901161, -139.70749999955297] ]. Possible malformed shape detected.",
testing-elasticsearch-761 | "at org.apache.lucene.geo.Tessellator.earcutLinkedList(Tessellator.java:455) ~[lucene-sandbox-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
testing-elasticsearch-761 | "at org.apache.lucene.geo.Tessellator.splitEarcut(Tessellator.java:592) ~[lucene-sandbox-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
testing-elasticsearch-761 | "at org.apache.lucene.geo.Tessellator.earcutLinkedList(Tessellator.java:453) ~[lucene-sandbox-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
testing-elasticsearch-761 | "at org.apache.lucene.geo.Tessellator.splitEarcut(Tessellator.java:592) ~[lucene-sandbox-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
testing-elasticsearch-761 | "at org.apache.lucene.geo.Tessellator.earcutLinkedList(Tessellator.java:453) ~[lucene-sandbox-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
testing-elasticsearch-761 | "at org.apache.lucene.geo.Tessellator.splitEarcut(Tessellator.java:592) ~[lucene-sandbox-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
testing-elasticsearch-761 | "at org.apache.lucene.geo.Tessellator.earcutLinkedList(Tessellator.java:453) ~[lucene-sandbox-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
testing-elasticsearch-761 | "at org.apache.lucene.geo.Tessellator.tessellate(Tessellator.java:114) ~[lucene-sandbox-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
testing-elasticsearch-761 | "at org.apache.lucene.document.LatLonShape.createIndexableFields(LatLonShape.java:67) ~[lucene-sandbox-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.GeoShapeIndexer$LuceneGeometryIndexer.visit(GeoShapeIndexer.java:1048) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.GeoShapeIndexer$LuceneGeometryIndexer.visit(GeoShapeIndexer.java:980) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.geometry.Polygon.visit(Polygon.java:96) ~[elasticsearch-geo-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.GeoShapeIndexer.indexShape(GeoShapeIndexer.java:195) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.GeoShapeIndexer.indexShape(GeoShapeIndexer.java:59) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.AbstractGeometryFieldMapper.parse(AbstractGeometryFieldMapper.java:438) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "... 17 more"] }
testing-elasticsearch-761 | {"type": "server", "timestamp": "2020-04-22T09:05:31,006Z", "level": "DEBUG", "component": "o.e.a.b.TransportShardBulkAction", "cluster.name": "testing-elasticsearch", "node.name": "master.elasticsearch", "message": "[ch-municipalities-test][0] failed to execute bulk item (index) index {[ch-municipalities-test][_doc][zaUioXEBhhvZ0_hagc-C], source[n/a, actual length: [3kb], max length: 2kb]}", "cluster.uuid": "qaWPq407TVaXX-EXhhP1rw", "node.id": "zTHeleceQzSfrJ7X8CGfhw" , 
testing-elasticsearch-761 | "stacktrace": ["org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [coordinates] of type [geo_shape]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.AbstractGeometryFieldMapper.parse(AbstractGeometryFieldMapper.java:446) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:488) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:505) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:418) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:395) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:112) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:71) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:267) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:793) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:770) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:742) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:267) [elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:157) [elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
testing-elasticsearch-761 | "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
testing-elasticsearch-761 | "at java.lang.Thread.run(Thread.java:830) [?:?]",
testing-elasticsearch-761 | "Caused by: java.lang.IllegalArgumentException: Unable to Tessellate shape [[24.0, 5.0] [-55.0, -75.0] [85.0, -56.0] [45.0, -135.0] [5.0, -175.0] [65.0, 5.0] [-15.0, 65.0] [-35.0, -115.0] [25.0, 55.0] [-61.0, -15.0] [-45.0, -75.0] [17.0, -175.0] [55.0, 145.0] [-16.0, 114.0] [-64.0, 54.0] [85.0, -105.0] [54.0, -135.0] [8.0, 142.0] [-35.0, 65.0] [-55.0, 5.0] [-65.0, 105.0] [-65.0, -92.0] [26.0, 65.0] [26.0, 45.0] [53.0, -45.0] [-65.0, -105.0] [75.0, -165.0] [-65.0, 125.0] [-57.0, 75.0] [5.0, 43.0] [-5.0, 43.0] [-65.0, 34.0] [15.0, -45.0] [-55.0, -45.0] [45.0, -6.0] [74.0, -6.0] [-13.0, -85.0] [-23.0, -136.0] [76.0, -136.0] [40.0, -89.0] [4.0, 2.0] [46.0, -47.0] [-45.0, -45.0] [27.0, -93.0] [65.0, -105.0] [-45.0, -173.0] [-58.0, 134.0] [5.0, 56.0] [-53.0, 1.0] [-85.0, 88.0] [74.0, -67.0] [-32.0, -136.0] [53.0, -118.0] [-64.0, 162.0] [-4.0, 121.0] [-29.0, -36.0] [-22.0, 42.0] [-69.0, 124.0] [-15.0, 56.0] [86.0, -117.0] [7.0, 109.0] [10.0, 90.0] [-15.0, 64.0] [-15.0, 4.0] [-5.0, -15.0] [76.0, 65.0] [65.0, 5.0] [5.0, -175.0] [12.0, 155.0] [59.0, 25.0] [22.0, 178.0] [22.0, -38.0] [-36.0, -63.0] [-7.0, -13.0] [17.0, 32.0] [60.0, 111.0] [35.0, 175.0] [-35.0, -105.0] [35.0, 25.0] [75.0, 65.0] [43.0, 97.0] [-35.0, 97.0] [15.0, 65.0] [75.0, 65.0] [-5.0, 5.0] [25.0, 5.0] [76.0, 105.0] [25.0, 105.0] [-86.0, 35.0] [74.0, -86.0] [-45.0, 163.0] [34.0, 163.0] [-16.0, -150.0] [-35.0, -145.0] [-85.0, -145.0] [-5.0, -125.0] [-55.0, -105.0] [-27.0, -66.0] [-25.0, -75.0] [-5.0, -66.0] [25.0, -66.0] [5.0, -55.0] [-45.0, -45.0] [35.0, -25.0] [-75.0, 74.0] [10.0, 113.0] [-53.0, 115.0] [-1.0, 94.0] [65.0, 75.0] [4.0, 75.0] [55.0, 75.0] [75.0, 85.0] [24.0, 85.0] [-35.0, 105.0] [-55.0, -155.0] [-65.0, 75.0] [-40.0, 113.0] [25.0, 145.0] [84.0, 160.0] [15.0, 175.0] [-3.0, 177.0] [-53.0, -178.0] [69.0, -7.0] [82.0, -66.0] [51.0, -2.0] [74.0, 98.0] [-29.0, 138.0] [-65.0, 145.0] [-25.0, 145.0] [85.0, 175.0] [45.0, -164.0] [35.0, -105.0] [-15.0, -26.0] [-55.0, -6.0] [45.0, 35.0] [25.0, 45.0] [15.0, 105.0] [-25.0, 125.0] [52.0, 123.0] [85.0, 125.0] [43.0, 117.0] [-65.0, 85.0] [-5.0, 55.0] [15.0, 55.0] [36.0, -137.0] [65.0, -137.0] [-75.0, -65.0] [-45.0, -45.0] [-6.0, -45.0] [75.0, -15.0] [35.0, 35.0] [55.0, 55.0] [75.0, 115.0] [27.0, 105.0] [-55.0, 95.0] [-5.0, 65.0] [-35.0, 65.0] [-23.0, 85.0] [-73.0, 80.0] [73.0, -65.0] [-46.0, 164.0] [-65.0, 135.0] [-45.0, 124.0] [43.0, 124.0] [65.0, -143.0] [-55.0, -143.0] [-25.0, -95.0] [25.0, 175.0] [-64.0, 135.0] [15.0, 94.0] [24.0, 5.0] ]. Possible malformed shape detected.",
testing-elasticsearch-761 | "at org.apache.lucene.geo.Tessellator.earcutLinkedList(Tessellator.java:455) ~[lucene-sandbox-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
testing-elasticsearch-761 | "at org.apache.lucene.geo.Tessellator.splitEarcut(Tessellator.java:592) ~[lucene-sandbox-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
testing-elasticsearch-761 | "at org.apache.lucene.geo.Tessellator.earcutLinkedList(Tessellator.java:453) ~[lucene-sandbox-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
testing-elasticsearch-761 | "at org.apache.lucene.geo.Tessellator.splitEarcut(Tessellator.java:592) ~[lucene-sandbox-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
testing-elasticsearch-761 | "at org.apache.lucene.geo.Tessellator.earcutLinkedList(Tessellator.java:453) ~[lucene-sandbox-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
testing-elasticsearch-761 | "at org.apache.lucene.geo.Tessellator.splitEarcut(Tessellator.java:592) ~[lucene-sandbox-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
testing-elasticsearch-761 | "at org.apache.lucene.geo.Tessellator.earcutLinkedList(Tessellator.java:453) ~[lucene-sandbox-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
testing-elasticsearch-761 | "at org.apache.lucene.geo.Tessellator.splitEarcut(Tessellator.java:592) ~[lucene-sandbox-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
testing-elasticsearch-761 | "at org.apache.lucene.geo.Tessellator.earcutLinkedList(Tessellator.java:453) ~[lucene-sandbox-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
testing-elasticsearch-761 | "at org.apache.lucene.geo.Tessellator.splitEarcut(Tessellator.java:592) ~[lucene-sandbox-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
testing-elasticsearch-761 | "at org.apache.lucene.geo.Tessellator.earcutLinkedList(Tessellator.java:453) ~[lucene-sandbox-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
testing-elasticsearch-761 | "at org.apache.lucene.geo.Tessellator.splitEarcut(Tessellator.java:592) ~[lucene-sandbox-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
testing-elasticsearch-761 | "at org.apache.lucene.geo.Tessellator.earcutLinkedList(Tessellator.java:453) ~[lucene-sandbox-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
testing-elasticsearch-761 | "at org.apache.lucene.geo.Tessellator.tessellate(Tessellator.java:114) ~[lucene-sandbox-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
testing-elasticsearch-761 | "at org.apache.lucene.document.LatLonShape.createIndexableFields(LatLonShape.java:67) ~[lucene-sandbox-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.GeoShapeIndexer$LuceneGeometryIndexer.visit(GeoShapeIndexer.java:1048) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.GeoShapeIndexer$LuceneGeometryIndexer.visit(GeoShapeIndexer.java:1035) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.GeoShapeIndexer$LuceneGeometryIndexer.visit(GeoShapeIndexer.java:980) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.geometry.MultiPolygon.visit(MultiPolygon.java:44) ~[elasticsearch-geo-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.GeoShapeIndexer.indexShape(GeoShapeIndexer.java:195) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.GeoShapeIndexer.indexShape(GeoShapeIndexer.java:59) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "at org.elasticsearch.index.mapper.AbstractGeometryFieldMapper.parse(AbstractGeometryFieldMapper.java:438) ~[elasticsearch-7.6.2.jar:7.6.2]",
testing-elasticsearch-761 | "... 17 more"] }

@michaelimfeld
Copy link

Whoops my reverse proxy was limiting the max request size. Maybe some sort of error message on the UI would be nice to indicate that the request wasn't successful.
Anyway got it working now, thanks for the support :)

Now I'm facing a different issue, the importer seems to mess up the encoding:
"Zürich" --> "Zürich". But I'll keep debugging and open up a new issue for this.

@jsanz
Copy link
Member Author

jsanz commented Apr 22, 2020

Now I'm facing a different issue, the importer seems to mess up the encoding:
"Zürich" --> "Zürich". But I'll keep debugging and open up a new issue for this.

That's already reported at #56604

You may want to try the OGR path as described on this blogpost, just as of today I answered another post adding some details on how to use it with Docker, since OGR/GDAL is still to officially release a version with support for Elasticsearch 7

@michaelimfeld
Copy link

Awesome works, thank you @jsanz !

@shayant85
Copy link

shayant85 commented Apr 29, 2020 •
edited
Loading

Hi all... I've got a sample countries json file from here https://datahub.io/core/geo-countries... ran into the hanging on "writing to index" issue while uploading it to Maps. Same issue after cleaning it with mapshare. Any tips would be appreciated.... using Kibana 7.6.2.

@jsanz
Copy link
Member Author

jsanz commented Apr 30, 2020

thanks for reporting @shayant85 these are my results after testing this dataset in particular:

  • This is a simple but detailed dataset, just two fields, 23MB
  • Coordinates have excessive decimals, you can use this command to remove unnecessary figures

mapshaper -i countries.geojson -o countries.mapshaper.geojson rfc7946

  • I could upload the dataset and all variations I did using mapshaper and ogr2ogr to a local installation without any issues
  • I could not upload the dataset or any variation to two different cloud instances 7.6.2 (1GB Kibana) or 7.7 (2GB Kibana)
  • I tried to scale the 7.6.2 Kibana instance to 8GB, same result.
  • I successfully imported the dataset in a cloud instance with a more aggressive simplification, so my assumption is that the problem may be sending a large payload, but not really sure
mapshaper -i countries.geojson -simplify visvalingam "60%" -o countries.mapshaper.geojson rfc7946
  • I successfully imported the dataset in a cloud instance using ogr2ogr (last version, using a Docker image), but got a warning about a feature crossing the dateline
$ docker run --rm --network=host -u $(id -u ${USER}):$(id -g ${USER}) \
-v /home:/home \
osgeo/gdal:alpine-small-latest \
ogr2ogr "ES:https://USER:PASSWORD@CLOUD_URL" \
-nln countries_ogr_cli /home/j/desktop/countries.geojson

Warning 1: At least one geometry has a bounding box outside of [-180,180] longitude range and/or [-90,90] latitude range. Undefined behaviour

@shayant85
Copy link

Thanks for the info @jsanz ... I think some of the issues we are seeing are also related to some stuff being cached in Kibana / Maps if you had an unsuccessful upload attempt (In my case unsuccessful refers to getting stuck on "writing to index" indefinitely).

For instance I've noticed that if after an unsuccessful attempt you simply click 'Cancel' and try to add a new layer again with the - presumably - fixed file from mapshare, then it will not work. However, if you either restart Kibana or refresh the browser (sometimes) the same fixed file will work. I had this happen with two files which I had fixed (AKA imported/exported) via mapshare so thought it is worth mentioning.

@nreese
Copy link
Contributor

nreese commented Apr 1, 2021

Closed by #92620.

Just tried loading leeds wards geojson file and it now successfully imports the file without any UI hangs or leaving the add layer button disabled. The import indexes 21 features and fails indexing 3 features.

@nreese nreese closed this as completed Apr 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kindsun kindsun

Labels
bug Fixes for quality problems that affect the customer experience [Deprecated-Use Team:Presentation]Team:Geo Former Team Label for Geo Team. Now use Team:Presentation
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@jsanz @nreese @kindsun @thomasneirynck @kevinkraft @michaelimfeld @elasticmachine @shayant85