Elasticsearch常用查询过滤接口与值得注意的问题「建议收藏」

Elasticsearch常用查询过滤接口与值得注意的问题「建议收藏」Elasticsearch常用查询过滤接口与值得注意的问题 简介 本文将介绍一些ES查询过滤的接口和一些值得问题。 在ES中主要是查询,并且只有在bool查询中才有过滤上下文,当然聚合函数中也可能出…

Elasticsearch常用查询过滤接口与值得注意的问题

Elasticsearch常用查询过滤接口与值得注意的问题

简介

本文将介绍一些ES查询过滤的接口和一些值得问题。

在ES中主要是查询,并且只有在bool查询中才有过滤上下文,当然聚合函数中也可能出现过滤上下文。

过滤不计算相关性评分,并且能够缓存,所以应该优先考虑过滤。

具体的做法就是使用bool查询的filter,后面会详细介绍。

关于查询过滤的rest api介绍,可以参考Elasticsearch查询过滤解惑

数据准备bulk

首先,我们使用bulk添加一些测试数据:

public class BulkTest {

    private static final String[] homes = {"河北省", "山西省", "辽宁省", "吉林省", "江苏省", "浙江省", "安徽省", "福建省", "江西省", "山东省", "河南省", "湖北省", "湖南省", "广东省", "海南省", "四川省", "贵州省", "云南省", "陕西省", "甘肃省", "青海省", "黑龙江省", "台湾省", "北京市", "天津市", "上海市", "重庆市", "广西壮族自治区", "西藏自治区", "宁夏回族自治区", "新疆维吾尔自治区", "内蒙古自治区", "香港特别行政区", "澳门特别行政区"};

    private RestHighLevelClient client;

    @Before
    public void setUp() {
        HttpHost host = new HttpHost("localhost", 9200, "http");
        client = new RestHighLevelClient(RestClient.builder(host));
    }

    @Test
    public void add() throws IOException {
        BulkRequest request = new BulkRequest();
        IndexRequest indexRequest;
        List<UserInfo> userInfos = UserInfo.getUserInfo(10000);
        String indexName = "user";
        for(UserInfo userInfo : userInfos){
            indexRequest = new IndexRequest(indexName).id(userInfo.id).source(JSON.toJSONString(userInfo), XContentType.JSON);
            request.add(indexRequest);
        }
        client.bulk(request, RequestOptions.DEFAULT);

    }

    private static class UserInfo{
        private String id;
        private Long createTime;
        private String createTimeStr;
        private short status;
        private String home;
        private String option;

        public static List<UserInfo> getUserInfo(int size){
            LinkedList<UserInfo> userInfos = new LinkedList<>();
            LocalDateTime now = LocalDateTime.now();
            Random random = new Random();
            int count = 1;
            for(int i=0;i<size;i++){
                UserInfo userInfo = new UserInfo();
                LocalDateTime localDateTime = now.plusDays(random.nextInt(1000));
                userInfo.setId(String.format("%s%05d",localDateTime.format(DatetimeUtil.YYYYMMDDHHMMSS_FORMATTER),count++));
                userInfo.setCreateTimeStr(localDateTime.format(DatetimeUtil.DATE_TIME_FORMATTER));
                userInfo.setCreateTime(DatetimeUtil.getLocalDateTimeMill(localDateTime));
                userInfo.setHome(homes[random.nextInt(homes.length)]);
                userInfo.setOption(homes[random.nextInt(homes.length)]);
                userInfo.setStatus((short) random.nextInt(10));
                userInfos.add(userInfo);
            }
            return userInfos;
        }

        public String getId() {
            return id;
        }

        public void setId(String id) {
            this.id = id;
        }

        public Long getCreateTime() {
            return createTime;
        }

        public void setCreateTime(Long createTime) {
            this.createTime = createTime;
        }

        public String getCreateTimeStr() {
            return createTimeStr;
        }

        public void setCreateTimeStr(String createTimeStr) {
            this.createTimeStr = createTimeStr;
        }

        public short getStatus() {
            return status;
        }

        public void setStatus(short status) {
            this.status = status;
        }

        public String getHome() {
            return home;
        }

        public void setHome(String home) {
            this.home = home;
        }

        public String getOption() {
            return option;
        }

        public void setOption(String option) {
            this.option = option;
        }
    }
}

bool查询

Java rest API中有一个QueryBuilders工厂类,可以创建各个查询bulider。

bool查询中最重要的是filter,表示过滤。

当然,也可以使用常见的must,表示必须满足, should,表示至少一个,must_not表示必须不。

下面直接上代码:

@Test
public void boolQueryBuilder() throws IOException {
    SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();

    BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
    boolQueryBuilder.filter(QueryBuilders.termQuery("status", "5"));//bool查询中过滤上下文

    searchSourceBuilder.query(boolQueryBuilder);
    searchSourceBuilder.from(0);//从0开始
    searchSourceBuilder.size(20);//默认10
    searchSourceBuilder.sort("createTime", SortOrder.DESC);//排序

    SearchRequest searchRequest = new SearchRequest(INDEX_NAME);
    searchRequest.source(searchSourceBuilder);
    SearchResponse search = client.search(searchRequest, RequestOptions.DEFAULT);
    SearchHits hits = search.getHits();
    for (SearchHit hit : hits) {
        System.out.println(hit.getSourceAsString());
    }
}

上面代码就是通过过滤的方式找status字段为5的文档,并且给了获取开始条数和获取多少条。

默认从0开始,获取10条,不要深度分页,需要深度分页参考后面的scroll和search after

当然也可以通过sort指定排序字段。

term与terms查询

term查询,表示精确匹配,terms和term基本一样,但是terms允许设置多个值,只要有一个值精确匹配就算匹配成功。

可以直接使用term查询,但是还是建议尽量放将term查询放在bool查询的filter中。

@Test
public void term() throws IOException {
    SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();

    TermQueryBuilder termQueryBuilder = QueryBuilders.termQuery("home.keyword", "四川省");
//        TermQueryBuilder termQueryBuilder = QueryBuilders.termQuery("home", "四川省");
    searchSourceBuilder.query(termQueryBuilder);
    System.out.println(searchSourceBuilder.toString());

    SearchRequest searchRequest = new SearchRequest(INDEX_NAME);
    searchRequest.source(searchSourceBuilder);

    SearchResponse search = client.search(searchRequest, RequestOptions.DEFAULT);
    SearchHits hits = search.getHits();
    for (SearchHit hit : hits) {
        System.out.println(hit.getSourceAsString());
    }
}

@Test
public void terms() throws IOException {
    BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
    TermsQueryBuilder termsQueryBuilder = QueryBuilders.termsQuery("status", 5,6);
    boolQueryBuilder.filter(termsQueryBuilder);
    SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
    searchSourceBuilder.query(boolQueryBuilder);

    SearchRequest searchRequest = new SearchRequest(INDEX_NAME);
    searchRequest.source(searchSourceBuilder);

    SearchResponse search = client.search(searchRequest, RequestOptions.DEFAULT);
    SearchHits hits = search.getHits();
    for (SearchHit hit : hits) {
        System.out.println(hit.getSourceAsString());
    }
}

注意: 如果使用明明有数据,但是使用term查询不出来,这个时候,也许你可以检查一下mapping对应的字段了。

很多朋友不喜欢设置mapping,或者设置了动态mapping,这样动态添加字符串类型的时候,ES就会自动生成一个text类型,并且设置fields,取前256字符设置为keyword。

这个时候,就要使用field-name.keyword这个字段来查询,而不是field-name字段。很多时候查询时间字符串不准确基本也是这个原因。

范围查询

范围查询非常简单,也非常常用:

@Test
public void range() throws IOException {
    RangeQueryBuilder rangeQueryBuilder = QueryBuilders.rangeQuery("status").gte(4).lte(7);

    SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
    searchSourceBuilder.query(rangeQueryBuilder);

    SearchRequest searchRequest = new SearchRequest(INDEX_NAME);
    searchRequest.source(searchSourceBuilder);
    SearchResponse search = client.search(searchRequest, RequestOptions.DEFAULT);
    SearchHits hits = search.getHits();
    for (SearchHit hit : hits) {
        System.out.println(hit.getSourceAsString());
    }
}

ids

通过id集合查询文档:

 @Test
public void ids() throws IOException {
    IdsQueryBuilder idsQueryBuilder = QueryBuilders.idsQuery().addIds("20210313173331","20211101173331");

    SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
    searchSourceBuilder.query(idsQueryBuilder);
    SearchRequest searchRequest = new SearchRequest(INDEX_NAME);

    searchRequest.source(searchSourceBuilder);
    SearchResponse search = client.search(searchRequest, RequestOptions.DEFAULT);
    SearchHits hits = search.getHits();
    for (SearchHit hit : hits) {
        System.out.println(hit.getSourceAsString());
    }
}

exists

查询存在指定字段的文档:

 @Test
public void exists() throws IOException {
    //检查字段是否存在
//        ExistsQueryBuilder existsQueryBuilder = QueryBuilders.existsQuery("status");
    ExistsQueryBuilder existsQueryBuilder = QueryBuilders.existsQuery("hello");
    SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
    searchSourceBuilder.query(existsQueryBuilder);

    SearchRequest searchRequest = new SearchRequest(INDEX_NAME);
    searchRequest.source(searchSourceBuilder);
    SearchResponse search = client.search(searchRequest, RequestOptions.DEFAULT);
    SearchHits hits = search.getHits();
    for (SearchHit hit : hits) {
        System.out.println(hit.getSourceAsString());
    }
}

match

@Test
public void match() throws IOException {
    MatchQueryBuilder matchQueryBuilder = QueryBuilders.matchQuery("home", "四川");

    SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
    searchSourceBuilder.query(matchQueryBuilder);

    SearchRequest searchRequest = new SearchRequest(INDEX_NAME);
    searchRequest.source(searchSourceBuilder);
    SearchResponse search = client.search(searchRequest, RequestOptions.DEFAULT);
    SearchHits hits = search.getHits();
    for (SearchHit hit : hits) {
        System.out.println(hit.getSourceAsString());
    }
}

multi_match

multi_match和match差不多,但是可以指定多个字段搜索。

@Test
public void multiMatch() throws IOException {
    MultiMatchQueryBuilder multiMatchQueryBuilder = QueryBuilders.multiMatchQuery("四川", "home", "option");

    SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
    searchSourceBuilder.query(multiMatchQueryBuilder);

    SearchRequest searchRequest = new SearchRequest(INDEX_NAME);
    searchRequest.source(searchSourceBuilder);
    SearchResponse search = client.search(searchRequest, RequestOptions.DEFAULT);
    SearchHits hits = search.getHits();
    for (SearchHit hit : hits) {
        System.out.println(hit.getSourceAsString());
    }
}

scroll

有时候,在做统计的时候,可能需要搜索全部数据,如果数据量非常大,需要深度分页,简单的查询可能就不行了,这个时候就需要scroll。

scroll之所以有效,是因为它不做全局排序,这样在query阶段这个节点只需要查询自己的数据集,返回满足条件的id集合就可以了。

scroll会维护这个id集合的上下文一段时间,这样就可以查询全量数据。

import org.apache.http.HttpHost;
import org.elasticsearch.action.search.ClearScrollRequest;
import org.elasticsearch.action.search.ClearScrollResponse;
import org.elasticsearch.action.search.SearchRequest;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.action.search.SearchScrollRequest;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.common.unit.TimeValue;
import org.elasticsearch.index.query.BoolQueryBuilder;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.index.query.RangeQueryBuilder;
import org.elasticsearch.search.Scroll;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.search.builder.SearchSourceBuilder;
import org.elasticsearch.search.fetch.subphase.FetchSourceContext;
import org.junit.Before;
import org.junit.Test;

import java.io.BufferedOutputStream;
import java.io.FileOutputStream;
import java.io.IOException;

public class ScrollTest {

    private RestHighLevelClient client;

    @Before
    public void setUp() {
        HttpHost host = new HttpHost("127.0.0.1", 9200, "http");
        client = new RestHighLevelClient(RestClient.builder(host));
    }

    @Test
    public void scroll() throws IOException {
        long start = System.currentTimeMillis();
        FileOutputStream fileOutputStream = new FileOutputStream("F:\tmp\long_scroll9.txt");
        BufferedOutputStream bufferedOutputStream = new BufferedOutputStream(fileOutputStream);
        final Scroll scroll = new Scroll(TimeValue.timeValueMinutes(1L));
        SearchRequest searchRequest = new SearchRequest("user");
        searchRequest.scroll(scroll);
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();

        //只获取指定字段
        String[] fields = {"id","createTime"};
        FetchSourceContext sourceContext = new FetchSourceContext(true,fields,null);
        searchSourceBuilder.fetchSource(sourceContext);

        BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
//        RangeQueryBuilder lastlogintime = QueryBuilders.rangeQuery("createTimeStr");
//        lastlogintime.gte("2019-08-11 00:00:00");

        RangeQueryBuilder lastlogintime = QueryBuilders.rangeQuery("createTime");
        lastlogintime.gte(1597298833674L);

        boolQueryBuilder.filter(lastlogintime);
//        boolQueryBuilder.must(lastlogintime);
        searchSourceBuilder.query(boolQueryBuilder);
//        searchSourceBuilder.sort("_doc");

        searchRequest.source(searchSourceBuilder);


        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
        String scrollId = searchResponse.getScrollId();
        int count = 0;
        SearchHit[] searchHits = searchResponse.getHits().getHits();
        count+=searchHits.length;
//        print(searchHits);
        print(bufferedOutputStream,searchHits);

        while (searchHits != null && searchHits.length > 0) {
            SearchScrollRequest scrollRequest = new SearchScrollRequest(scrollId);
            scrollRequest.scroll(scroll);
            searchResponse = client.scroll(scrollRequest, RequestOptions.DEFAULT);
            scrollId = searchResponse.getScrollId();
            searchHits = searchResponse.getHits().getHits();
            count+=searchHits.length;
//            print(searchHits);
            print(bufferedOutputStream,searchHits);
        }
        bufferedOutputStream.close();
        System.out.println(count);
        ClearScrollRequest clearScrollRequest = new ClearScrollRequest();
        clearScrollRequest.addScrollId(scrollId);
        ClearScrollResponse clearScrollResponse = client.clearScroll(clearScrollRequest, RequestOptions.DEFAULT);
        boolean succeeded = clearScrollResponse.isSucceeded();
        System.out.println(succeeded);
        System.out.println(System.currentTimeMillis() - start);
    }

    private static void print(BufferedOutputStream bufferedOutputStream, SearchHit[] searchHits) throws IOException {
        for (SearchHit hit : searchHits) {
            bufferedOutputStream.write(hit.getSourceAsString().getBytes());
            bufferedOutputStream.write("
".getBytes());
        }
    }

    private static void print(SearchHit[] searchHits){
        for(SearchHit searchHit : searchHits){
            System.out.println(searchHit.getSourceAsString());
        }
    }
}

search after

scroll也有自己的局限,例如在query阶段满足条件的ids特别多,整个过程就会变得非常慢。

这个时候就可以考虑使用search after,search after和scroll原理基本一样,不过search after是实时的。

import org.apache.http.HttpHost;
import org.elasticsearch.action.search.SearchRequest;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.index.query.BoolQueryBuilder;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.index.query.RangeQueryBuilder;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.search.builder.SearchSourceBuilder;
import org.junit.Before;
import org.junit.Test;

import java.io.BufferedOutputStream;
import java.io.FileOutputStream;
import java.io.IOException;

public class SearchAfterTest {

    private static RestHighLevelClient client;

    @Before
    public void setUp() {
        HttpHost host = new HttpHost("127.0.0.1", 9200, "http");
        client = new RestHighLevelClient(RestClient.builder(host));
    }

    @Test
    public void search() throws IOException {
        long start = System.currentTimeMillis();
        Object[] objects = null;
        FileOutputStream fileOutputStream = new FileOutputStream("F:\tmp\safter_long_filter6.txt");
        BufferedOutputStream bufferedOutputStream = new BufferedOutputStream(fileOutputStream);
        boolean type = true;
        while (type) {
            SearchHit[] hits = searchAfter(client, objects);
            if(hits.length == 0){
                break;
            }
            objects = hits[hits.length-1].getSortValues();
            if (hits.length < 1000) {
                type = false;
            }
            writeData(bufferedOutputStream,hits);
        }
        bufferedOutputStream.close();
        System.out.println(System.currentTimeMillis() - start);
        client.close();
    }

    private static void writeData(BufferedOutputStream bufferedOutputStream, SearchHit[] searchHits) throws IOException {
        for (SearchHit hit : searchHits) {
            bufferedOutputStream.write(hit.getSourceAsString().getBytes());
            bufferedOutputStream.write("
".getBytes());
        }
    }

    public static SearchHit[] searchAfter(RestHighLevelClient client, Object[] objects) throws IOException {
        SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();

        BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();

//        RangeQueryBuilder lastlogintime = QueryBuilders.rangeQuery("createTimeStr");
        RangeQueryBuilder lastlogintime = QueryBuilders.rangeQuery("createTimeStr.keyword");
        lastlogintime.gte("2020-08-11 00:00:00");
//        RangeQueryBuilder lastlogintime = QueryBuilders.rangeQuery("createTime");
//        lastlogintime.gte(1597298833674L);
//        lastlogintime.gte(1597298833674L).lte(1597298833674L);
        boolQueryBuilder.filter(lastlogintime);

        sourceBuilder.query(boolQueryBuilder);
        sourceBuilder.size(1000);
//        sourceBuilder.sort("_id", SortOrder.DESC);
        if(objects != null) {
            sourceBuilder.searchAfter(objects);
        }
        SearchRequest searchRequest = new SearchRequest();
        searchRequest.indices("user");
        searchRequest.source(sourceBuilder);
        SearchResponse response = client.search(searchRequest, RequestOptions.DEFAULT);
        SearchHit[] hits = response.getHits().getHits();
        return hits;
    }
}

版权声明:本文内容由互联网用户自发贡献,该文观点仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 举报,一经查实,本站将立刻删除。
转载请注明出处: https://daima100.com/6750.html

(0)
上一篇 2023-04-04
下一篇 2023-04-04

相关推荐

  • mysql系列(十)——Mysql常见的工具

    mysql系列(十)——Mysql常见的工具一、压力测试工具——【mysqlslap】 二、 数据库表物理结构修改工具——【pt-online-schema-change】 三、慢查询分析工具—— 【mysqldumpslow】 四、慢查询分…

    2023-03-20
    154
  • Python Chdir:改变当前工作目录的方法

    Python Chdir:改变当前工作目录的方法Chdir是Python中提供的一个功能强大的方法,能够改变Python程序的工作目录。在Python中,每个正在运行的程序都有一个当前工作目录,即操作系统当前默认的工作目录。使用Chdir方法可以方便地改变程序的当前工作目录,便于进行文件的读写操作等。

    2024-04-06
    74
  • mysql 5.7密码策略修改「终于解决」

    mysql 5.7密码策略修改「终于解决」
    mysql 5.7密码策略修改 1、查看当前的密码策略 show variables like ‘validate_password%’; 2、密码策略的各…

    2023-04-03
    171
  • mysql性能问题「终于解决」

    mysql性能问题「终于解决」
    性能优化的目的是什么呢?(合理利用可利用的资源) 性能优化的方向: 硬件方面:CPU、内存、IO、网络、硬盘、显卡 软件方面:mysql这个软件内部优化,比…

    2023-04-06
    155
  • docker compose实战_Docker

    docker compose实战_Docker一、 概述 compose 是用来定义和运行一个或多个容器(通常都是多个)运行和应用的工具。使用 compose 可以简化容器镜像的构建以及容器的运行。 compose 使用 YAML 文件来定义多容

    2023-05-20
    135
  • mysql事务隔离级别介绍和演示「建议收藏」

    mysql事务隔离级别介绍和演示「建议收藏」什么是事务 事务是应用程序中一系列对数据库的操作,所有操作必须成功完成,否则在每个操作中所作的所有更改都会被撤消。也就是事务具有原子性,一个事务中的一系列的操作要么全部成功,要么一个都不做。 事务的结

    2023-03-19
    148
  • Python实现按钮制作功能

    Python实现按钮制作功能按钮是一种常见的交互元素,可以在图形用户界面(GUI)中使用。在 Python 中,我们可以使用多种库来创建不同类型的按钮。本文将介绍一些常见的 Python 按钮制作库,并为您提供每种库的代码示例。让我们一起来看看吧!

    2024-03-21
    76
  • 为什么你需要混沌工程以及 Chaos Mesh®「终于解决」

    为什么你需要混沌工程以及 Chaos Mesh®「终于解决」信心的毁灭与重建 在我最开始学习编程的时候,我一直觉得写程序是很简单的事情,程序总是按照我的想法串行的执行,给一个输入,总是有着符合预期的固定输出。那时候写代码,可能大的挑战在于理解分支,循环,但无…

    2023-03-20
    158

发表回复

您的电子邮箱地址不会被公开。 必填项已用*标注