MS RFC 121:MapCache维度中的PostgreSQL和ElasticSearch支持

日期

2018-06-25

作者

杰罗姆·布伊

版本

地图缓存待定

1。概述

mapcache维度管理在sqlite文件中提供二级维度的动态存储，请参见 Second Level Dimensions 在里面 瓷砖集尺寸 . 此建议描述了对在PostgreSQL数据库或ElasticSearch索引中存储二级维度的支持。此外，时间维度作为第二级维度实施，它还将受益于这些新的维度后端。提出了一种新的配置接口来显式地选择时间维度后端。

2。建议的增强功能

2.1.PostgreSQL维度

由于两者都基于SQL，PostgreSQL维度与SQLite维度非常相似。从配置的角度来看， <list_query> and <validate_query> elements are identical to the ones of SQLite Dimension configuration. The <dbfile> element (path of SQLite file containing dimension values) is replaced by a <connection> element which specifies connection information to the PostgreSQL database containing dimension values. This element contains the string required by the PQconnectdb() Libpq-C库的函数。

<!-- PostgreSQL Dimension -->
<dimension type="postgresql" name="sensor" default="phr">

    <!-- Access Point -->
    <connection>
        host=localhost user=mapcache password=mapcache dbname=mapcache port=5433
    </connection>

    <!-- All-values Query -->
    <list_query>
        SELECT distinct(product) FROM dim
    </list_query>

    <!-- Selected-values Query -->
    <validate_query>
        SELECT product FROM dim
        WHERE sensor=:dim
    </validate_query>

</dimension>

2.2。弹性搜索维度

ElasticSearch与SQLite和PostgreSQL的主要区别是它的查询语言Query-DSL基于JSON而不是SQL。所以呢， <list_query> 和 <validate_query> 元素应包含以该语言表达的查询。在XML配置中插入JSON数据可能需要将其封装在 <![CDATA[ ... ]]> 元素。

此外，ElasticSearch响应的复杂结构（也用JSON表示）也提出了从查询响应中获取子维度值的提取指令需求。

例如，假设此ElasticSearch查询：

{   "size": 0,
    "aggs": { "distinct_product": { "terms": { "field": "product/keyword" } } },
    "query": { "term": { "sensor": "phr" } }
}

返回此响应：

{   "took": 0,
    "timed_out": false,
    "_shards": { "total": 5, "successful": 5, "skipped": 0, "failed": 0 },
    "hits": { "total": 5, "max_score": 0, "hits": [] },
    "aggregations": {
        "distinct_product": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
                { "key": "phr_img1", "doc_count": 1 },
                { "key": "phr_img2", "doc_count": 1 },
                { "key": "phr_img3", "doc_count": 1 }
            ]
        }
    }
}

然后，预期的子维度值为：

[ "phr_img1", "phr_img2", "phr_img3" ]

该列表是通过提取 aggregations 外词典中的条目，然后 distinct_product ，那么 buckets 。最后是 key 项目将从每个词典中提取 bucket 单子。提取指令采用类似路径的关键字列表的形式，用于从ElasticSearch提供的JSON响应中提取子维值：

[ "aggregations", "distinct_product", "buckets", "key" ]

这是通过新的XML元素在MapCache配置文件中指定的，即 <list_response> 和 <validate_response> ，分别包含提取指令 <list_query> 和 <validate_query> 查询。

ElasticSearch索引的访问点配置了 <http> 元素来代替SQLite的 <dbfile> 或PostgreSQL的 <connection> 。在那 <http> 元素，两者都是 <url> 和 <Content-Type> 须予指明。

<!-- ElasticSearch Dimension -->
<dimension type="elasticsearch" name="sensor" default="phr">

    <!-- Access Point -->
    <http>
        <url>http://localhost:9200/sensor/_search</url>
        <headers>
            <Content-Type>application/json</Content-Type>
        </headers>
    </http>

    <!-- All-values Query -->
    <list_query><![CDATA[
        {   "size": 0,
            "aggs": {
                "products": { "terms": { "field": "product.keyword" } }
            }
        }
    ]]></list_query>

    <!-- All-values Response Extration Instructions -->
    <list_response>
        [ "aggregations", "products", "buckets", "key" ]
    </list_response>

    <!-- Selected-values Query -->
    <validate_query><![CDATA[
        {   "size": 0,
            "aggs": {
                "products": { "terms": { "field": "product.keyword" } }
            },
            "query": { "term": { "sensor": ":dim" } }
        }
    ]]></validate_query>

    <!-- Selected-values Response Extration Instructions -->
    <validate_response>
        [ "aggregations", "products", "buckets", "key" ]
    </validate_response>

</dimension>

2.3。时间维度

时间维度当前使用SQLite后端存储允许的维度值。以下是当前时间维度配置的示例：

 <!-- "time" dimension, "old-style" definition

      This example defines a "time" dimension whose possible values are
      stored in the /data/times.sqlite SQLite file. The default value is
      "d1".

      A WMS request with that dimension may contain, e.g.
      "... &time=1999-08-11T11:03:07Z/2001-09-21T08:17:56Z& ..."
 -->
 <dimension type="time" name="time" default="d1">
   <dbfile>/data/times.sqlite</dbfile>
   <query>
        SELECT strftime("%Y-%m-%dT%H:%M:%SZ",ts) FROM timedim
         WHERE ts &gt;= datetime(:start_timestamp,"unixepoch")
           AND ts &lt;= datetime(:end_timestamp,"unixepoch")
      ORDER BY ts DESC
   </query>
 </dimension>

从Dimensions后端将以除SQLite以外的其他方式实现的角度来看，使Time Dimensions后端不可知可能是相关的。从配置的角度来看，这是通过添加布尔值来实现的 time 属性添加到下列任意类型的维度 sqlite ， postgresql 或 elasticsearch 。以下是此类配置的示例。

 <!-- "time" dimension, "new-style" definition using a SQLite back-end

      This example defines a "time" dimension whose possible values are
      stored in the /data/times.sqlite SQLite file. The default value is
      "d1".

      A WMS request with that dimension may contain, e.g.
      "... &time=1999-08-11T11:03:07Z/2001-09-21T08:17:56Z& ..."
 -->
 <dimension type="sqlite" name="time" default="d1" time="true">
   <dbfile>/data/times.sqlite</dbfile>
   <list_query>SELECT ts FROM timedim</list_query>
   <validate_query>
        SELECT strftime("%Y-%m-%dT%H:%M:%SZ",ts) FROM timedim
         WHERE ts &gt;= datetime(:start_timestamp,"unixepoch")
           AND ts &lt;= datetime(:end_timestamp,"unixepoch")
      ORDER BY ts DESC
   </validate_query>
 </dimension>

 <!-- "time" dimension, "new-style" definition using a PostgreSQL back-end

      This example defines a "time" dimension whose possible values are
      stored in the postgresql://mapcache:mapcache@localhost:5433/time
      PostgreSQL database. The default value is "d1".

      A WMS request with that dimension may contain, e.g.
      "... &time=1999-08-11T11:03:07Z/2001-09-21T08:17:56Z& ..."
 -->
 <dimension type="postgresql" name="time" default="d1" time="true">
   <connection>
     host=localhost user=mapcache password=mapcache dbname=times port=5433
   </connection>
   <list_query>SELECT ts FROM timedim</list_query>
   <validate_query>
        SELECT to_char(ts,'YYYY-MM-DD"T"HH24:MI:SS"Z"') FROM timedim
         WHERE ts &gt;= to_timestamp(:start_timestamp)
           AND ts &lt;= to_timestamp(:end_timestamp)
      ORDER BY ts DESC
   </validate_query>
 </dimension>

 <!-- "time" dimension, "new-style" definition using an ElasticSearch back-end

      This example defines a "time" dimension whose possible values are
      stored in the http://localhost:9200/times ElasticSearch index. The
      default value is "d1".

      A WMS request with that dimension may contain, e.g.
      "... &time=1999-08-11T11:03:07Z/2001-09-21T08:17:56Z& ..."
 -->
 <dimension type="elasticsearch" name="time" default="d1" time="true">
   <http>
     <url>http://localhost:9200/times/_search</url>
     <headers>
       <Content-Type>application/json</Content-Type>
     </headers>
   </http>

   <list_query><![CDATA[
     { "query": { "match_all": { } } }
   ]]></list_query>
   <list_response><![CDATA[
     [ "hits", "hits", "_source", "ts" ]
   ]]></list_response>

   <validate_query><![CDATA[
     { "query": {
         "range": {
           "ts": { "gte": :start_timestamp, "lte": :end_timestamp }
         }
       },
       "sort": { "ts": { "order": "desc" } },
       "script_fields": {
         "iso_ts": {
           "lang": "painless",
           "source": "SimpleDateFormat f=new SimpleDateFormat(params.fmt); f.setTimeZone(params.tz); return (f.format(doc.ts.value.getMillis()))",
           "params": {
             "fmt": "yyyy-MM-dd'T'HH:mm:ss'Z'",
             "tz: "UTC"
           }
         }
       }
     }
   ]]></validate_query>
   <validate_response><![CDATA[
     [ "hits", "hits", "fields", "iso_ts", 0 ]
   ]]></validate_response>

 </dimension>

时间维度与隐式SQLite后端的兼容性问题

与显式后端配置一起，时间维度的几乎向后兼容配置仍然提供隐式SQLite后端。以下示例突出显示了这些差异：

<!-- "time" dimension This example defines a "time" dimension whose possible values are stored in the /data/times.sqlite SQLite file. The default value is "2010". A WMS request with that dimension may contain, e.g. "... &time=1999-08-11T11:03:07Z/2001-09-21T08:17:56Z& ..." --> <dimension type="time" name="time" default="2010"> <dbfile>/data/times.sqlite</dbfile> <query> SELECT strftime("%Y-%m-%dT%H:%M:%SZ",ts) FROM time WHERE ts &gt;= datetime(:start_timestamp,"unixepoch") AND ts &lt;= datetime(:end_timestamp,"unixepoch") ORDER BY ts DESC </query> </dimension>

<!-- "time" dimension This example defines a "time" dimension whose possible values are stored in the /data/times.sqlite SQLite file. The default value is "2010". A WMS request with that dimension may contain, e.g. "... &time=1999-08-11T11:03:07Z/2001-09-21T08:17:56Z& ..." --> <dimension type="time" name="time" default="2010"> <dbfile>/data/times.sqlite</dbfile> <validate_query> SELECT strftime("%Y-%m-%dT%H:%M:%SZ",ts) FROM time WHERE ts &gt;= datetime(:start_timestamp,"unixepoch") AND ts &lt;= datetime(:end_timestamp,"unixepoch") ORDER BY ts DESC </validate_query> <list_query>SELECT ts FROM time</list_query> </dimension>

到mapcache 1.6.1

从MapCache>1.6.1打开

三。实施细节

3.1。依赖关系

PostgreSQL和ElasticSearch维度都需要特定的库才能工作。这两种需求都是通过现成的解决方案来满足的。PostgreSQL接口使用第三方库，而ElasticSearch接口使用嵌入到MapCache中的代码。

PostgreSQL

libpq 是PostgreSQL的C API。这是一个第三方库，应该链接到mapcache，以便PostgreSQL维度后端工作。

ElasticSearch

虽然ElasticSearch JSON查询可以作为简单的文本字符串传递到服务器，但必须解析ElasticSearch JSON响应才能提取相关信息，以便由MapCache进一步处理。为此，提出了现成的解决方案，即 cJSON ，可以通过麻省理工学院的许可证在GitHub上获得。档案 cJSON.h 和 cJSON.c 简单地复制到MapCache项目中，分别位于 include/ 和 lib/ 目录。

3.2。受影响的文件

文件名	状态	描述
include/mapcache.h	被改进的	由提议的特性引入的接口更改
lib/dimension.c	被改进的	此文件中只保留所有维度后端通用的处理。
lib/dimension_sqlite.c，lib/dimension_es.c，lib/dimension_pg.c	新的	后端特定的进程获取自己的源文件。
lib/dimension_time.c	新的	时间维度获取自己的源文件
包括/cjson.h、lib/cjson.c	新的	Elsaticsearch后端的嵌入式JSON解析器
cmakelists.txt，include/mapcache-config.h.in	被改进的	可选外部PostgreSQL库的帐户
lib/util.c	被改进的	从建议的功能中获得新的辅助需求
lib/service_wms.c，lib/tileset.c，lib/configuration_xml.c，util/mapcache_seed.c	被改进的	拟议特征对现有规范的各种影响
mapcache.xml	被改进的	PostgreSQL维度的配置示例

3.3。文档

MAPCACHE的 瓷砖集尺寸 文件应相应更新。