Package org.machanism.machai.gw.processor

Class AIFileProcessor

java.lang.Object
org.machanism.machai.project.ProjectProcessor
org.machanism.machai.gw.processor.AbstractFileProcessor
org.machanism.machai.gw.processor.AIFileProcessor
Direct Known Subclasses:
ActProcessor, GuidanceProcessor
public class AIFileProcessor extends AbstractFileProcessor
File processor that drives a configured ProcessProvider provider with project-aware context, prompt metadata, optional external prompt inclusions, public configuration substitution, and function-tool registration.

The processor can handle a single file, a project folder, or a path/pattern scan. Before a prompt is sent to the provider it is normalized, include markers are resolved recursively, public configuration placeholders are substituted, project layout details are stored for project-context tools, and processing metadata is supplied to the provider as JSON. The processing metadata is generated by getProcessInfo(ProjectLayout, File) and includes the processed file's project-relative path, the interactive or non-interactive processing mode, and the operating-system name. This metadata is serialized as JSON and is supplied as provider context so tools and model prompts can identify the file and execution environment being processed.

Supported special markers and parameters

  • FILE_INCLUDED_MARKER: a line prefix used to include UTF-8 content from http://, https://, or file:// references. Included content is parsed again, so includes may be nested.
  • EXIT_SPECIAL_PROMPT_COMMAND: in interactive mode, entering this command exits processing successfully.
  • CONTINUE_SPECIAL_PROMPT_COMMAND: in interactive mode, entering this command accepts the current provider response and continues without another provider prompt.
  • ENABLED_TOOLS_PARAM_NAME: YAML front-matter property used to limit the provider tools enabled for the current prompt.
  • PUBLIC_PROP_GROUP_NAME: configuration-property prefix whose values are exposed for substitution in prompts, for example ${public.projectName}.

Supported prompt input parameters

Prompts may start with YAML front matter delimited by ---. Supported properties include gw.model, which overrides the configured provider or model for the prompt, and enabledTools, which may be either a scalar or a YAML list naming tools to enable. String values in the metadata are resolved with the active configurator before use.

Examples


 AIFileProcessor processor = new AIFileProcessor(rootDir, configurator, "openai:gpt-4.1");
 processor.setInstructions("Follow the project coding standards.");
 processor.process(projectLayout, file, ">>> file://docs/review-prompt.md\nReview the project.");
 

 ---
 gw.model: ${public.reviewModel}
 enabledTools:
   - get_project_context_variable
   - read_file
 ---
 Analyze ${public.projectName} and use
 ``` 
 >>> file://docs/checklist.md
 ``` 
 as checklist input.

  • Field Details

    • FRONT_MATTER_MARKER

      private static final String FRONT_MATTER_MARKER
      Delimiter that marks the beginning and end of YAML prompt front matter.
      See Also:

    • logger

      private static final org.slf4j.Logger logger
      Logger for processing lifecycle, diagnostic, and included-content messages.

    • ENABLED_TOOLS_PARAM_NAME

      public static final String ENABLED_TOOLS_PARAM_NAME
      Parameter name used to configure the list of tool definitions exposed to the LLM agent during an episode.

      This parameter is typically declared as YAML metadata inside an episode's head block. It accepts a comma-separated list of tool names to selectively restrict the agent's toolset for that specific stage. If omitted or left empty, all registered tools remain available to the agent.

      Example Usage

      1. Restrict tools inside an Episode YAML head block: 
        
     ---
     enabledTools: 
     	- get_bindex
     	- pick_libraries
     ---
     # Episode Instructions
     Locate and validate our integration boundaries...
      See Also:

    • PUBLIC_PROP_GROUP_NAME

      public static final String[] PUBLIC_PROP_GROUP_NAME
      Prefix for property groups whose values are exposed and injectable directly into prompt templates.

      Any configuration property starting with this prefix (e.g., public.schemaUrl) will be automatically collected and made available as a fully-prefixed variable placeholder (e.g., ${public.schemaUrl}) inside LLM prompt files or system instructions.

      Example Usage

      1. Define the property in a configuration file (e.g., mcp.properties): 
        
     public.schemaUrl=https://raw.githubusercontent.com/machanism-org/bindex/schema-v2.json
     public.projectName=Bindex Core
      2. Reference the property in an Act prompt template: 
        
     # Load Bindex Schema
     Validate your JSON against this schema: ${public.schemaUrl}
     Processing context for: ${public.projectName}

    • FILE_INCLUDED_MARKER

      public static final String FILE_INCLUDED_MARKER
      Prefix marker for prompt lines that include external content. A line beginning with this marker is treated as a reference. Supported references are http://..., https://..., and file://....

      For file:// references, the path is resolved relative to the active project directory.

      Referenced content is read as UTF-8 and recursively parsed, so included files may contain additional include markers.

      Example: >>> file://docs/instructions.md (resolves to <projectDir>/docs/instructions.md)

      See Also:

    • LOG_OUTPUT_PREFIX

      public static final String LOG_OUTPUT_PREFIX
      The format pattern used to prepend logger output blocks when reporting responses generated by the AI provider.
      See Also:

    • EXIT_SPECIAL_PROMPT_COMMAND

      public static final String EXIT_SPECIAL_PROMPT_COMMAND
      Interactive-mode command that terminates processing immediately by throwing a ProcessTerminationException with exit code 0.

      Example interactive input: .

      See Also:

    • CONTINUE_SPECIAL_PROMPT_COMMAND

      public static final String CONTINUE_SPECIAL_PROMPT_COMMAND
      Interactive-mode command that accepts the current response and continues processing without sending another prompt to the provider.

      Example interactive input: >

      See Also:

    • NO_INTERACTIVE_SPECIAL_PROMPT_COMMAND

      private static final String NO_INTERACTIVE_SPECIAL_PROMPT_COMMAND
      Interactive-mode command that accepts the current response and continues processing in a non-interactive mode, running to completion without prompting the user again.

      Example interactive input: >>

      See Also:

    • model

      private String model
      The specific AI model identifier or provider label being utilized for operations.

    • instructions

      private String instructions
      Base instructions that set the persona, tone, and scope for the AI provider during execution.

    • defaultPrompt

      private String defaultPrompt
      Default fallback prompt used to process directories or projects when no specific file prompt is given.

    • interactive

      private boolean interactive
      Flag indicating whether interactive mode is active, allowing continuous prompt looping and execution.

    • functionTools

      protected List<FunctionTools> functionTools
      Collection of user-configured function tools that are registered with and made available to the AI provider.

    • functionToolsLoader

      private static FunctionToolsLoader functionToolsLoader
      Loader utility responsible for discovering, instantiating, and applying functional tools to the AI provider.

  • Constructor Details

    • AIFileProcessor

      public AIFileProcessor(File rootDir, Configurator configurator, String genai)
      Creates a processor for the given project directory and AI provider identifier.
      Parameters:
      rootDir - the project root directory
      configurator - the application configuration
      genai - the AI provider or model identifier

  • Method Details

    • process

      public String process(ProjectLayout projectLayout, File file, String prompt)
      Processes the given file using the currently configured instructions.
      Parameters:
      projectLayout - the current project layout metadata
      file - the file to process
      prompt - the prompt to send to the AI provider
      Returns:
      the provider response, or null when no response is produced

    • process

      protected String process(ProjectLayout projectLayout, File file, String instructions, String... prompts)
      Processes a specified file within a project layout by configuring and invoking a Generative AI (GenAI) provider using a sequence of instructions and prompts.

      This method performs the following operations:

      • Establishes the thread context for the provided ProjectLayout.
      • Extracts input parameters from the provided array of prompts.
      • Resolves the GenAI model configuration (falling back to the default configured model if not explicitly overridden in prompt metadata via gw.model).
      • Instantiates the target ProcessProvider provider and registers enabled toolkits and custom function tools.
      • Constructs system instructions by combining default bundle instructions with any custom parameters passed to instructions.
      • Feeds file-specific contextual metadata and substituted prompts to the AI provider.
      • Executes the AI operation and returns the generated content.

      Supported Input Properties (extracted dynamically from the prompts' metadata):

      • gw.model (String) - Overrides the default model identifier used to initialize the GenAI provider. If not present, the method falls back to the default instance model.
      • enabledTools (String or List<?>) - Configures which toolkits or tools should be enabled for the AI provider. Defined via the constant ENABLED_TOOLS_PARAM_NAME.
      • Any additional YAML key/value properties are copied into the layered configurator and can therefore participate in configuration substitution; they do not otherwise change provider execution.
      • Other YAML properties are retained as prompt configuration values and string values may be resolved through the active configurator before they are used. Properties not recognized by the processor are available for configuration substitution but do not otherwise alter processing.
      Parameters:
      projectLayout - the directory structure and metadata context of the active project
      file - the target file currently being processed
      instructions - additional custom system instructions to append to the default system instructions; can be null or blank
      prompts - a variable-length list or array of user prompt sequences to be evaluated and sent to the GenAI model
      Returns:
      the output string containing the model's response if processing was successful; null if prompts were empty or blank
      Throws:
      IllegalArgumentException - if the resolved GenAI model identifier is missing or no matching provider can be initialized

    • applyTools

      protected void applyTools(String instructions, String[] prompts, ProcessProvider provider, String[] tools)
      Registers the selected discovered tools and the tools explicitly added to this processor with the provider for the current request.
      Parameters:
      instructions - the resolved system instructions
      prompts - the resolved prompts
      provider - the provider that receives the tools
      tools - the selected tool names, or null for the default set

    • getEnabledTools

      private String[] getEnabledTools(Map<String,Object> inputProps, LayeredConfigurator conf)
      Resolves the tool names enabled for the current processing request.

      Prompt front matter takes precedence over the configured value. A string is split on whitespace, commas, and semicolons; a YAML list is converted to an array of item strings. Other value types are treated as though no explicit tool selection was supplied. When neither source provides a value, null is returned so the provider can use its default tool set.

      Parameters:
      inputProps - prompt parameters extracted from YAML front matter
      conf - layered configuration used as the fallback source
      Returns:
      enabled tool names, or null when no selection is configured

    • extractInputParams

      private String extractInputParams(String prompt, Map<String,Object> inputProps)
      Extracts supported YAML front-matter input parameters from the beginning of a prompt.

      A prompt may start with a YAML block delimited by ---. The block is removed from the returned prompt content and each YAML entry is merged into inputProps. String values are resolved with the processor configurator; non-string values, such as YAML lists, are preserved.

      Supported special input parameters include:

      • gw.model: model or provider identifier to use for this prompt.
      • enabledTools: a string or YAML list naming provider tools that should be enabled.

      Example: 

      
 ---
 gw.model: ${public.ai.model}
 enabledTools:
   - project-context
   - file-system
 ---
 Analyze this file.
      Parameters:
      prompt - the input prompt string which may contain a leading YAML configuration block
      inputProps - the map to populate with extracted and resolved parameters
      Returns:
      the stripped prompt content when a configuration block is present; otherwise, the original prompt string

    • removeFrontMatterData

      static String removeFrontMatterData(String prompt)
      Removes a leading, complete YAML front-matter block from a prompt.

      The prompt is trimmed before inspection. If it does not begin with the front-matter delimiter, or does not contain a closing delimiter, its trimmed content is returned unchanged.

      Parameters:
      prompt - the prompt from which leading front matter is removed; must not be null
      Returns:
      the prompt content after the closing delimiter, or the trimmed input when no complete leading front matter exists

    • resolveInputParamValue

      private Object resolveInputParamValue(Object value)
      Resolves configuration placeholders in a parsed front-matter value.

      Strings are resolved against this processor's configurator. Lists are copied with the same resolution applied recursively to each element; other YAML value types are returned unchanged.

      Parameters:
      value - the parsed YAML value, which may be null
      Returns:
      the resolved string or list, or the original non-string value

    • processModulesMultiThreaded

      void processModulesMultiThreaded(ProjectLayout projectLayout, List<String> modules)
      Processes the project modules in a multi-threaded manner using the specified layout and module list.

      If interactive mode is currently enabled, it is automatically disabled with a warning log, as interactive mode is not supported during concurrent multi-threaded execution.

      Overrides:
      processModulesMultiThreaded in class AbstractFileProcessor
      Parameters:
      projectLayout - the layout of the project containing configuration and structure details
      modules - the list of module identifiers or names to be processed

    • getProcessInfo

      public String getProcessInfo(ProjectLayout projectLayout, File file)
      Generates a structured JSON string containing execution metadata about the file being processed and the current processing environment context.

      This method builds a map structure containing processing details and attempts to serialize it to a JSON format. The generated map holds the following properties:

      • "PROCESSED_FILE_REL_PATH" - The relative path of the processed file with respect to the project directory.
      • "PROCESS_MODE" - The current interaction mode, returning "INTERACTIVE" if the execution is interactive, otherwise "NOT-INTERACTIVE".
      • "OS_NAME" - The operating-system name reported by the runtime, allowing consumers of the processing metadata to identify the execution environment.

      Serialization Fallback: If Jackson's ObjectMapper fails to serialize the map to a standard JSON string, the method falls back to the default string representation of the map (via Object.toString()).

      Parameters:
      projectLayout - the layout configuration of the project, used to resolve the base project directory for path relative-ization; must not be null
      file - the file currently undergoing processing, used to determine its relative path; must not be null
      Returns:
      a string representation of the processing information map; ideally a valid JSON-formatted string, or a stringified map representation if serialization fails
      See Also:

    • perform

      private String perform(File file, ProcessProvider provider)
      Executes the provider and, when interactive mode is enabled, processes the next user command or follow-up prompt.

      The exit command terminates processing, the continue command returns the current response, and the non-interactive command disables further input. Any other input is submitted to the provider and processing recurses until a command is received or input is unavailable.

      Parameters:
      file - the file associated with the current processing operation
      provider - the configured provider to execute and, if needed, receive a follow-up prompt
      Returns:
      the current or final provider response
      Throws:
      ProcessTerminationException - if the interactive exit command is entered

    • input

      protected String input()
      Obtains the next input from an interactive user session.

      The default implementation does not read from standard input because this processor may run without a supported interactive console. Subclasses can override this method to provide console or UI input.

      Returns:
      the next command or prompt, or null when input is unavailable

    • setProjectLayoutContext

      private void setProjectLayoutContext(ProjectLayout projectLayout)
      Extracts context metadata from the provided ProjectLayout and registers it in the project context registry.

      This method evaluates essential environment information (such as the operating system), project structure configurations (like name, IDs, and relative paths), and maps directory definitions (such as source files, tests, documentation, and sub-modules) into a centralized storage registry managed by ProjectContextFunctionTools.

      Directory collections are consolidated into formatted string information lines relative to the project directory before registration.

      Parameters:
      projectLayout - the ProjectLayout containing the current project structure, directories, and parent configurations; must not be null
      Throws:
      IllegalArgumentException - if an error occurs during JSON serialization or parsing of the layout information (wraps JsonProcessingException)
      See Also:

    • getDirInfoLine

      com.fasterxml.jackson.databind.node.ArrayNode getDirInfoLine(Collection<String> sources, File projectDir)
      Returns a JsonNode (ArrayNode) containing the names of directories from the given collection that exist within the specified project directory. Each directory name is wrapped in backticks.
      Parameters:
      sources - a collection of directory names (relative to projectDir) to check for existence
      projectDir - the base directory in which to check for the existence of each source directory
      Returns:
      a JsonNode (ArrayNode) of existing directory names, each wrapped in backticks (e.g., ["`src`", "`resources`"])

    • setInstructions

      public void setInstructions(String instructions)
      Sets the base instructions used for processing after normalizing line content and resolving supported references.
      Parameters:
      instructions - the raw instruction text

    • getInstructions

      public String getInstructions()
      Returns the current base instructions used for processing.
      Returns:
      the configured instruction text

    • parseLines

      public String parseLines(String data, File projectDir, Configurator conf)
      Normalizes multi-line input and resolves supported line references such as HTTP URLs and file: references.
      Parameters:
      data - the input text to parse
      projectDir - the project root used to resolve relative file references
      conf - the configurator used to substitute public properties
      Returns:
      the normalized text

    • tryToGetFromReference

      String tryToGetFromReference(String data, File projectDir, Configurator conf) throws IOException
      Resolves a single instruction line that may point to external content.

      If the instruction contains an external reference marker, it will be fetched and resolved. URLs starting with http:// or https:// are fetched remotely. URIs starting with file:// are resolved and loaded relative to the provided project directory.

      Parameters:
      data - the instruction line to inspect
      projectDir - the root directory of the project, used as the base context to resolve relative file:// references
      conf -
      Returns:
      the resolved content, or the original line when no reference is found, or null when the input is null
      Throws:
      IOException - if the referenced remote content or local file cannot be read

    • readFromHttpUrl

      static String readFromHttpUrl(String urlString) throws IOException
      Reads UTF-8 text content from the given HTTP or HTTPS URL.
      Parameters:
      urlString - the URL to read
      Returns:
      the response content as text
      Throws:
      IOException - if the URL cannot be read

    • readFromFilePath

      String readFromFilePath(String filePath, File projectDir) throws IOException
      Reads UTF-8 text content from the given file path.
      Parameters:
      filePath - the absolute or project-relative file path
      projectDir - the project root used to resolve a relative path
      Returns:
      the file content as text
      Throws:
      IOException - if the file cannot be opened or read

    • scanDocuments

      public void scanDocuments(File projectDir, String path) throws IOException
      Configures scanning based on the provided directory or path pattern and then starts scanning the project folder.

      The path argument may be specified as:

      • An absolute path — used as-is to scan a specific location.
      • A relative path — resolved against projectDir.
      • A glob pattern — e.g., "glob:**&#47;*.java", matched against files under projectDir.
      • A regex pattern — e.g., "regex:.*\\.java", matched against files under projectDir.
      If path equals the absolute path of projectDir, the entire project directory is scanned without applying any pattern matching.
      Parameters:
      projectDir - the project root directory; must not be null
      path - the directory, relative path, glob pattern, or regex pattern used to match files to scan; must not be blank
      Throws:
      IllegalArgumentException - if projectDir is null or path is blank
      IOException - if scanning fails

    • parsePath

      String parsePath(File projectDir, String path)
      Resolves the effective scan directory and converts it into a glob expression when required.
      Parameters:
      projectDir - the base project directory
      path - the configured scan directory
      Returns:
      the resolved path matcher expression
      Throws:
      IllegalArgumentException - if the resolved path is outside the project directory

    • getDefaultPrompt

      protected String getDefaultPrompt()
      Returns the default prompt used for folder processing.
      Returns:
      the default prompt

    • setDefaultPrompt

      protected void setDefaultPrompt(String defaultPrompt)
      Sets the default prompt used for folder processing.
      Parameters:
      defaultPrompt - the default prompt text

    • processFolder

      public void processFolder(ProjectLayout projectLayout)
      Processes the project root folder using the configured default prompt.
      Overrides:
      processFolder in class AbstractFileProcessor
      Parameters:
      projectLayout - the current project layout metadata

    • getModel

      public String getModel()
      Returns the configured AI model or provider identifier.
      Returns:
      the model or provider identifier

    • setModel

      public void setModel(String genai)
      Sets the AI model or provider identifier.
      Parameters:
      genai - the model or provider identifier

    • setInteractive

      public void setInteractive(boolean interactive)
      Enables or disables interactive processing mode.
      Parameters:
      interactive - true to enable interactive mode; otherwise false

    • isInteractive

      public boolean isInteractive()
      Indicates whether interactive processing mode is enabled.
      Returns:
      true when interactive mode is enabled; otherwise false

    • addTool

      public void addTool(FunctionTools toolFunction)
      Adds a tool definition that will be exposed to the AI provider.
      Parameters:
      toolFunction - the tool definition to add