analysis-ik README

The project's README from github.com/infinilabs/analysis-ik. As one markdown file: README.md.

IK Analysis for Elasticsearch and OpenSearch

Test License

The IK Analysis plugin integrates Lucene IK analyzer, and support customized dictionary. It supports major versions of Elasticsearch and OpenSearch. Maintained and supported with ❤️ by INFINI Labs.

The plugin comprises analyzer: ik_smart , ik_max_word, and tokenizer: ik_smart , ik_max_word

How to Install

🚀 Get the Packaged Plugins

You can download the packaged plugins from here:

https://release.infinilabs.com/

🛠️ Install via CLI

Alternatively, you can use the plugin CLI to install the plugin. Here's how:

For Elasticsearch:

bin/elasticsearch-plugin install https://get.infini.cloud/elasticsearch/analysis-ik/9.1.4

For OpenSearch:

bin/opensearch-plugin install https://get.infini.cloud/opensearch/analysis-ik/2.12.0

⚠️ Tip

Make sure to replace the version number with the one that matches your Elasticsearch or OpenSearch version.

Getting Started

1.create a index

curl -XPUT http://localhost:9200/index

2.create a mapping

curl -XPOST http://localhost:9200/index/_mapping -H 'Content-Type:application/json' -d'
{
        "properties": {
            "content": {
                "type": "text",
                "analyzer": "ik_max_word",
                "search_analyzer": "ik_smart"
            }
        }

}'

3.index some docs

curl -XPOST http://localhost:9200/index/_create/1 -H 'Content-Type:application/json' -d'
{"content":"美国留给伊拉克的是个烂摊子吗"}
'

curl -XPOST http://localhost:9200/index/_create/2 -H 'Content-Type:application/json' -d'
{"content":"公安部：各地校车将享最高路权"}
'

curl -XPOST http://localhost:9200/index/_create/3 -H 'Content-Type:application/json' -d'
{"content":"中韩渔警冲突调查：韩警平均每天扣1艘中国渔船"}
'

curl -XPOST http://localhost:9200/index/_create/4 -H 'Content-Type:application/json' -d'
{"content":"中国驻洛杉矶领事馆遭亚裔男子枪击 嫌犯已自首"}
'

4.query with highlighting

curl -XPOST http://localhost:9200/index/_search  -H 'Content-Type:application/json' -d'
{
    "query" : { "match" : { "content" : "中国" }},
    "highlight" : {
        "pre_tags" : ["<tag1>", "<tag2>"],
        "post_tags" : ["</tag1>", "</tag2>"],
        "fields" : {
            "content" : {}
        }
    }
}
'

Result

{
    "took": 14,
    "timed_out": false,
    "_shards": {
        "total": 5,
        "successful": 5,
        "failed": 0
    },
    "hits": {
        "total": 2,
        "max_score": 2,
        "hits": [
            {
                "_index": "index",
                "_type": "fulltext",
                "_id": "4",
                "_score": 2,
                "_source": {
                    "content": "中国驻洛杉矶领事馆遭亚裔男子枪击 嫌犯已自首"
                },
                "highlight": {
                    "content": [
                        "<tag1>中国</tag1>驻洛杉矶领事馆遭亚裔男子枪击 嫌犯已自首 "
                    ]
                }
            },
            {
                "_index": "index",
                "_type": "fulltext",
                "_id": "3",
                "_score": 2,
                "_source": {
                    "content": "中韩渔警冲突调查：韩警平均每天扣1艘中国渔船"
                },
                "highlight": {
                    "content": [
                        "均每天扣1艘<tag1>中国</tag1>渔船 "
                    ]
                }
            }
        ]
    }
}

Dictionary Configuration

Config file IKAnalyzer.cfg.xml can be located at {conf}/analysis-ik/IKAnalyzer.cfg.xml or {plugins}/elasticsearch-analysis-ik-*/config/IKAnalyzer.cfg.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">
<properties>
	<entry key="ext_dict">custom/mydict.dic;custom/single_word_low_freq.dic</entry>
	<entry key="ext_stopwords">custom/ext_stopword.dic</entry>
	<entry key="remote_ext_dict">location</entry>
	<entry key="remote_ext_stopwords">http://xxx.com/xxx.dic</entry>
</properties>

Hot-reload Dictionary

The current plugin supports hot reloading dictionary for IK Analysis, through the configuration mentioned earlier in the IK configuration file.

	<entry key="remote_ext_dict">location</entry>
	<entry key="remote_ext_stopwords">location</entry>

Among which location refers to a URL, such as http://yoursite.com/getCustomDict. This request only needs to meet the following two points to complete the segmentation hot update.

  1. The HTTP request needs to return two headers, one is Last-Modified, and the other is ETag. Both of these are of string type, and if either changes, the plugin will fetch new segmentation to update the word library.

  2. The content format returned by the HTTP request is one word per line, and the newline character is represented by \n.

Meeting the above two requirements can achieve hot word updates without the need to restart the ES instance.

You can place the hot words that need to be automatically updated in a .txt file encoded in UTF-8. Place it under nginx or another simple HTTP server. When the .txt file is modified, the HTTP server will automatically return the corresponding Last-Modified and ETag when the client requests the file. You can also create a separate tool to extract relevant vocabulary from the business system and update this .txt file.

FAQs

  1. Why isn't the custom dictionary taking effect?

Please ensure that the text format of your custom dictionary is UTF8 encoded.

  1. What is the difference between ik_max_word and ik_smart?

ik_max_word: Performs the finest-grained segmentation of the text. For example, it will segment "中华人民共和国国歌" into "中华人民共和国,中华人民,中华,华人,人民共和国,人民,人,民,共和国,共和,和,国国,国歌", exhaustively generating various possible combinations, suitable for Term Query.

ik_smart: Performs the coarsest-grained segmentation of the text. For example, it will segment "中华人民共和国国歌" into "中华人民共和国,国歌", suitable for Phrase queries.

Note: ik_smart is not a subset of ik_max_word.

Community

Fell free to join the Discord server to discuss anything around this project:

https://discord.gg/4tKTMkkvVX

License

Copyright ©️ INFINI Labs.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

analysis-ik (infinilabs/analysis-ik)

Searching Database…

17.5K ·GitHub
Analysis-ik
What is this?A plugin for Elasticsearch and OpenSearch that integrates the Lucene IK analyzer, adding Chinese word segmentation with support for customized dictionaries.
Is this a standalone search engine, or something I install into one?It is not standalone — it is a plugin that installs into your Elasticsearch or OpenSearch cluster and adds two analyzers/tokenizers, ikmaxword and iksmart, which you select in your index mapping.
How do I install it into my cluster?Download the packaged plugin matching your cluster version from the releases page, or install it via the plugin CLI (elasticsearch-plugin install / opensearch-plugin install) pointing at the matching release zip.
Which versions of Elasticsearch and OpenSearch does it support?It supports major versions of both, but the plugin build you install must match your exact cluster version number.
What trips people up when adding a custom dictionary?Dictionary files must be UTF-8 encoded or they will not take effect, and a hot-reload word source must return Last-Modified and ETag headers for updates to be fetched.
Does it run on my machine, or inside my search cluster?Inside the cluster — it works wherever Elasticsearch or OpenSearch runs, typically self-hosted Linux servers or Docker deployments.
What is the difference between ikmaxword and iksmart?ikmaxword performs the finest-grained segmentation, exhaustively generating all token combinations — suited to Term Query. iksmart performs the coarsest segmentation, suited to phrase queries; it is not a subset of ikmaxword.
Can I update the word dictionary without restarting the cluster?Yes — hot reload. Point a location URL in IKAnalyzer.cfg.xml at a UTF-8 word list (one word per line, for example a .txt served by nginx); when its Last-Modified or ETag changes, the plugin fetches the new words automatically.
What are similar projects?zincsearch, OpenSearch, elasticsearch, flexsearch, typesense, sonic, bleve, druid
Which topics is it in?analyzer, word, dictionary, analysis, segmentation, full-text-search, elasticsearch, search-engine, nlp, java, plugin, free
Creator
infinilabs
Size
15.2 MB
Stars
17.5K
Updated
CommitJun 30ReleaseLatestMay 06
Age
14y
Commits · 90 Days