Multimodal Extraction

Documentation Index

Fetch the complete documentation index at: https://docs.openmemind.com/llms.txt

Use this file to discover all available pages before exploring further.

​

Overview

​

Supported content types

​

Plugin setup

​

Maven dependencies

<dependency>
  <groupId>com.openmemind.ai</groupId>
  <artifactId>memind-plugin-rawdata-document</artifactId>
  <version>${memind.version}</version>
</dependency>

<dependency>
  <groupId>com.openmemind.ai</groupId>
  <artifactId>memind-plugin-rawdata-image</artifactId>
  <version>${memind.version}</version>
</dependency>

<dependency>
  <groupId>com.openmemind.ai</groupId>
  <artifactId>memind-plugin-rawdata-audio</artifactId>
  <version>${memind.version}</version>
</dependency>

<dependency>
  <groupId>com.openmemind.ai</groupId>
  <artifactId>memind-plugin-rawdata-toolcall</artifactId>
  <version>${memind.version}</version>
</dependency>

​

Runtime registration

var memory = Memory.builder()
    // other runtime components: chat client, store, vector, text search...
    .rawDataPlugin(new DocumentRawDataPlugin())
    .build();

var memory = Memory.builder()
    // other runtime components...
    .rawDataPlugin(new DocumentRawDataPlugin())
    .rawDataPlugin(new ImageRawDataPlugin())
    .rawDataPlugin(new AudioRawDataPlugin())
    .rawDataPlugin(new ToolCallRawDataPlugin())
    .build();

​

Spring Boot starters

​

Extraction modes

​

Extract conversation messages

• examples and tests
• importing a transcript
• processing a finished conversation
• extracting memory from a known context window

var result = memory.addMessages(
    memoryId,
    messages,
    ExtractionConfig.defaults().withLanguage("Chinese")
).block();

​

Extract streaming messages

memory.addMessage(memoryId, Message.user("Can you review this design?")).block();

memory.addMessage(memoryId, Message.assistant("The main issue is that the API boundary is unclear."))
    .block();

​

Commit buffered context

• a session ends
• an agent run finishes
• the application is shutting down
• you want to make sure the latest buffered context is written

var result = memory.commit(memoryId).block();

memory.addMessage(memoryId, userMessage).block();
memory.addMessage(memoryId, assistantMessage).block();

// At the end of the session or run:
memory.commit(memoryId).block();

​

Extract documents

var request = DocumentExtractionRequests.document(memoryId, documentContent);

var result = memory.extract(request).block();

• project docs
• runbooks
• design documents
• meeting notes
• release notes
• support articles
• internal knowledge bases

​

Extract images

var request = ImageExtractionRequests.image(memoryId, imageContent);

var result = memory.extract(request).block();

• UI screenshots
• architecture diagrams
• error screenshots
• visual bug reports
• whiteboard notes
• product or design references

​

Extract audio

var request = AudioExtractionRequests.audio(memoryId, audioContent);

var result = memory.extract(request).block();

• transcript segments
• speaker or timing metadata when available
• summarized captions
• source references
• extracted facts, events, preferences, and decisions

​

Extract tool calls

var result = ToolCallMemories.report(memory, memoryId, toolCalls).block();

• coding agents
• research agents
• automation agents
• test and deployment agents
• long-running task agents

• tool experience
• durable directives
• reusable playbooks
• resolved problems
• failed attempts
• successful workflows

​

Extract custom raw content

• internal business events
• workflow records
• logs
• incident reports
• CRM records
• product analytics events
• domain-specific artifacts

var result = memory.extract(
    memoryId,
    rawContent,
    ExtractionConfig.defaults()
).block();

​

Configure extraction

var config = ExtractionConfig.defaults().withLanguage("Chinese");

var result = memory.addMessages(memoryId, messages, config).block();

​

Response format

ExtractionResult
  memoryId
  rawDataResult
  memoryItemResult
  insightResult
  status
  duration
  errorMessage
  insightPending

​

Inspect extracted memory

​

Best practices

• Use addMessages() for complete conversation segments.
• Use addMessage() for real-time agents and chatbots.
• Use commit() at session end or before shutdown.
• Use extract() for documents, images, audio, tool calls, and custom raw content.
• Use plugin helpers when available.

• Conversation extraction works with memind-core.
• Document extraction needs the document raw-data plugin.
• Image extraction needs the image raw-data plugin.
• Audio extraction needs the audio raw-data plugin.
• Tool-call extraction needs the tool-call raw-data plugin.

• Keep useful metadata such as source client, content type, timestamps, file names, URLs, and tool names.
• Prefer source-aware content objects over plain text when the source matters.
• Inspect Raw Data captions before tuning item extraction.

• Do not force every single chat message into memory as a complete unit.
• Let streaming messages accumulate until there is enough context.
• Commit the final buffer when a run ends.

• Start with default extraction behavior.
• Review Raw Data, Memory Items, Threads, and Insights in Memind UI.
• Adjust language, chunking, plugin options, or insight behavior only when the output shows a real need.