Tweaking the ingest pipeline parser

Your ingest pipeline execution context may have more, or less, processors available, or you could have written your own ingest processors running on the target Elasticsearch instance; see Writing Your Own Ingest Processor For Elasticsearch for more information.

If this is your case, this guide is for you!

Making your own parser instance

In any case, you must make your own parser instance with your custom list of processors. For this, two options are available:

• You can derive the default Elasticsearch instance, add your custom processors and remove processors you don’t want available, by using IngestPipelineParser.copy(). For example:

  from estceque.builtin import DEFAULT_INGEST_PIPELINE_PARSER
  
  my_parser = DEFAULT_INGEST_PIPELINE_PARSER.copy(
      # `with_processors` can be set with the processors you want to add
      # or replace. This parameter is optional.
      with_processors={"example": ExampleProcessor, ...},
  
      # `without_processors` can be set with a sequence of names
      # representing the processors you want to remove compared to the
      # default parser. This parameter is optional.
      without_processors=["pipeline", ...],
  )
  

• You can start from scratch by defining your own parser instance, using IngestPipelineParser directly, and add the processors you want. For example:

  from estceque.core import IngestPipelineParser
  
  my_parser = IngestPipelineParser(processors={
      "example": ExampleProcessor,
      ...
  })
  

You can then use this parser with both the validation and parsing functions, using the parser keyword parameter:

validate_ingest_pipeline(..., parser=my_parser)

Creating a custom processor

You can make a custom processor by making a class inheriting from Processor, which defines the common properties for all Elasticsearch processors.

For example, an example processor could be defined by this snippet:

from estceque.core import Processor

class ExampleProcessor(Processor):
    hello: str