数据库（4）ElasticSearch基础

2016-07-22

一、ES安装

docker安装

1	$ docker run -d --name elasticsearch -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" elasticsearch:7.10.2

镜像地址：https://hub.docker.com/_/elasticsearch，启动后访问http://localhost:9200/

参数调整

#1、修改/etc/security/limits.conf，最后添加：
soft nofile 65536
hard nofile 131072
soft nproc 2048
hard nproc 4096
soft memlock unlimited
hard memlock unlimited

#2、修改/etc/sysctl.conf，最后添加：
vm.max_map_count = 262144
$ sysctl -p

二、ES使用

基本信息查询

健康信息查询：GET /_cat/health?v
节点信息查询：GET /_cat/nodes?v
分片信息查询：GET /_cat/indices?v

索引操作（可以认为是数据库）

添加索引：PUT /INDEX_NAME?pretty
查看索引：GET /_cat/indices?v
删除索引：DELETE /INDEX_NAME?pretty

文档操作（行操作）

添加文档（指定id=1），也可用于覆盖。doc为type（可认为是数据表），可自定义。

PUT /INDEX_NAME/doc/1?pretty
{
  "name": "iherr"
}

添加文档（不指定id）

POST /INDEX_NAME/doc?pretty
{
  "name": "iherr"
}
# 返回结果中_id为随机返回的id
{
  "_index": "iherr",
  "_type": "doc",
  "_id": "N6e6VG8Bq8d_C4rwMnJR",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  },
  "_seq_no": 3,
  "_primary_term": 1
}

修改文档

POST /INDEX_NAME/doc/1/_update?pretty
{
  "doc": { "age": 18 }
}

POST /INDEX_NAME/doc/1/_update?pretty
{
  "script" : "ctx._source.age += 5"
}

删除文档：

1	DELETE /INDEX_NAME/doc/2?pretty

批处理

使用_bulk的API。

POST /INDEX_NAME/doc/_bulk?pretty
{"index":{"_id":"1"}}
{"name": "iherr1" }
{"index":{"_id":"2"}}
{"name": "iherr2" }

POST /INDEX_NAME/doc/_bulk?pretty
{"update":{"_id":"1"}}
{"doc": { "name": "iherr" } }
{"delete":{"_id":"2"}}

搜索

ES搜索的DSL，推荐使用GET+ 请求Body方式，符合规范但少见，一般我个人喜欢用POST方法代替。

# 默认查询
POST /bank/_search

# 按account_number字段正序排列
POST /bank/_search?q=*&sort=account_number:asc&pretty

# 按account_number字段正序排列，分页查询from和size，并只取account_number和balance这2个字段数据
POST /bank/_search
{
  "query": { "match_all": {} },
  "_source": ["account_number", "balance"],
  "sort": [
    { "account_number": "asc" }
  ],
  "from":10,
  "size":2
}

# 查询account_number为20的数据
POST /bank/_search
{
  "query": { "match": { "account_number": 20 } }
}

# 查询address包含mill的数据
POST /bank/_search
{
  "query": { "match": { "address": "mill" } }
}

# 查询address包含mill或者lane的数据
POST /bank/_search
{
  "query": { "match": { "address": "mill lane" } }
}

# 查询address包含"mill lane"整体字段的数据
POST /bank/_search
{
  "query": { "match_phrase": { "address": "mill lane" } }
}

bool query示例：

# 交集 
POST /bank/_search
{
 "query": {
    "bool": {
      "must": [
        { "match": { "address": "mill" } },
        { "match": { "address": "lane" } }
      ]
    }
  }
}

# 并集 
POST /bank/_search
{
 "query": {
    "bool": {
      "should": [
        { "match": { "address": "mill" } },
        { "match": { "address": "lane" } }
      ]
    }
  }
}

# neither "mill" nor "lane"
POST /bank/_search
{
  "query": {
    "bool": {
      "must_not": [
        { "match": { "address": "mill" } },
        { "match": { "address": "lane" } }
      ]
    }
  }
}

# 40岁且不在ID州的
POST /bank/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "age": "40" } }
      ],
      "must_not": [
        { "match": { "state": "ID" } }
      ]
    }
  }
}

# 使用filter，balance在20000-30000之间
POST /bank/_search
{
  "query": {
    "bool": {
      "must": { "match_all": {} },
      "filter": {
        "range": {
          "balance": {
            "gte": 20000,
            "lte": 30000
          }
        }
      }
    }
  }
}

三、java的demo

pom.xml

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>2.2.1.RELEASE</version>
    <relativePath/>
</parent>

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
    </dependency>
    <dependency>
        <groupId>org.projectlombok</groupId>
        <artifactId>lombok</artifactId>
    </dependency>
</dependencies>

properties文件

1
2
3

spring.elasticsearch.rest.uris= http://127.0.0.1:9200
#spring.elasticsearch.rest.username=elastic
#spring.elasticsearch.rest.password=123456

Person.java

@Data
@NoArgsConstructor
@AllArgsConstructor
@Accessors(chain = true)
@Document(indexName = "doc",type = "person")
public class Person {

    @Id
    private String id;

    @Field(type = FieldType.Text)
    private String name;
}

PersonRepository.java

1 2	public interface PersonRepository extends ElasticsearchRepository<Person,String> { }

PersonController.java

@RestController
@Slf4j
public class PersonController {
    @Autowired
    PersonRepository personRepository;

    @GetMapping(value = "/test1")
    public String test1(String id){
        personRepository.save(new Person(id,"测试"+id));
        return "add success";
    }

    @GetMapping(value="test2")
    public String test2(String id){
        Optional<Person> resu  = personRepository.findById(id);
        Person person = resu.get();
        log.info(person.getName());
        return person.getName();
    }

    @GetMapping(value = "/test3")
    public List<Person> test3(){
        Iterable<Person> result = personRepository.findAll();
        Iterator<Person> resu = result.iterator();
        List<Person> list = new ArrayList<>();
        while (resu.hasNext()){
            list.add(resu.next());
        }
        return list;
    }

    @GetMapping(value = "/test4")
    public void test4(String id){
        personRepository.deleteById(id);
    }

    @GetMapping(value = "/test5")
    public void test5(){
        personRepository.deleteAll();
    }
}

测试

也可直接访问网址查看：http://localhost:9200/doc/person/_search

四、ELK

https://hub.docker.com/r/sebp/elk

使用文档 https://hub.docker.com/r/sebp/elk

http://www.dockerchina.cn/?id=64 基于Docker快速搭建ELK、Kafka

https://blog.csdn.net/qq_36236890/article/details/80445651 Spring Cloud ELK+kafka日志分析平台(一) 搭建

https://mp.weixin.qq.com/s/nOpITzYx9upTfSNENP5zug 2021 版本 ElasticSearch 学习手册！

https://www.toutiao.com/i6769823502758314503 搭建ELK容器化，so easy