Package org.machanism.machai.gw.processor

Class AIFileProcessor

java.lang.Object
org.machanism.machai.project.ProjectProcessor
org.machanism.machai.gw.processor.AbstractFileProcessor
org.machanism.machai.gw.processor.AIFileProcessor
Direct Known Subclasses:
ActProcessor, GuidanceProcessor
public class AIFileProcessor extends AbstractFileProcessor
File processor that drives a configured Genai provider with project-aware context, prompt metadata, optional external prompt inclusions, public configuration substitution, and function-tool registration.

The processor can handle a single file, a project folder, or a path/pattern scan. Before a prompt is sent to the provider it is normalized, include markers are resolved recursively, public configuration placeholders are substituted, project layout details are stored for project-context tools, and processing metadata is supplied to the provider as JSON. The processing metadata is generated by getProcessInfo(ProjectLayout, File) and includes the processed file's project-relative path, the interactive or non-interactive processing mode, and the operating-system name. This metadata is serialized as JSON and is supplied as provider context so tools and model prompts can identify the file and execution environment being processed.

Supported special markers and parameters

  • FILE_INCLUDED_MARKER: a line prefix used to include UTF-8 content from http://, https://, or file:// references. Included content is parsed again, so includes may be nested.
  • EXIT_SPECIAL_PROMPT_COMMAND: in interactive mode, entering this command exits processing successfully.
  • CONTINUE_SPECIAL_PROMPT_COMMAND: in interactive mode, entering this command accepts the current provider response and continues without another provider prompt.
  • ENABLED_TOOLS_PARAM_NAME: YAML front-matter property used to limit the provider tools enabled for the current prompt.
  • PUBLIC_PROP_GROUP_NAME: configuration-property prefix whose values are exposed for substitution in prompts, for example ${public.projectName}.

Supported prompt input parameters

Prompts may start with YAML front matter delimited by ---. Supported properties include gw.model, which overrides the configured provider or model for the prompt, and enabledTools, which may be either a scalar or a YAML list naming tools to enable. String values in the metadata are resolved with the active configurator before use.

Examples


 AIFileProcessor processor = new AIFileProcessor(rootDir, configurator, "openai:gpt-4.1");
 processor.setInstructions("Follow the project coding standards.");
 processor.setDefaultPrompt(">>> file://docs/review-prompt.md\nReview the project.");
 processor.processFolder(projectLayout);
 

 ---
 gw.model: ${public.reviewModel}
 enabledTools:
   - get_project_context_variable
   - read_file
 ---
 Analyze ${public.projectName} and use
 ``` 
 >>> file://docs/checklist.md
 ``` 
 as checklist input.

  • Field Details

    • FRONT_MATTER_MARKER

      private static final String FRONT_MATTER_MARKER
      See Also:

    • logger

      private static final org.slf4j.Logger logger

    • ENABLED_TOOLS_PARAM_NAME

      public static final String ENABLED_TOOLS_PARAM_NAME
      Parameter name used to configure the list of tool definitions exposed to the LLM agent during an episode.

      This parameter is typically declared as YAML metadata inside an episode's head block. It accepts a comma-separated list of tool names to selectively restrict the agent's toolset for that specific stage. If omitted or left empty, all registered tools remain available to the agent.

      Example Usage

      1. Restrict tools inside an Episode YAML head block: 
        
     ---
     enabledTools: 
     	- get_bindex
     	- pick_libraries
     ---
     # Episode Instructions
     Locate and validate our integration boundaries...
      See Also:

    • PUBLIC_PROP_GROUP_NAME

      public static final String[] PUBLIC_PROP_GROUP_NAME
      Prefix for property groups whose values are exposed and injectable directly into prompt templates.

      Any configuration property starting with this prefix (e.g., public.schemaUrl) will be automatically collected and made available as a fully-prefixed variable placeholder (e.g., ${public.schemaUrl}) inside LLM prompt files or system instructions.

      Example Usage

      1. Define the property in a configuration file (e.g., mcp.properties): 
        
     public.schemaUrl=https://raw.githubusercontent.com/machanism-org/bindex/schema-v2.json
     public.projectName=Bindex Core
      2. Reference the property in an Act prompt template: 
        
     # Load Bindex Schema
     Validate your JSON against this schema: ${public.schemaUrl}
     Processing context for: ${public.projectName}

    • FILE_INCLUDED_MARKER

      public static final String FILE_INCLUDED_MARKER
      Prefix marker for prompt lines that include external content. A line beginning with this marker is treated as a reference. Supported references are http://..., https://..., and file://....

      For file:// references, the path is resolved relative to the active project directory.

      Referenced content is read as UTF-8 and recursively parsed, so included files may contain additional include markers.

      Example: >>> file://docs/instructions.md (resolves to <projectDir>/docs/instructions.md)

      See Also:

    • LOG_OUTPUT_PREFIX

      public static final String LOG_OUTPUT_PREFIX
      The format pattern used to prepend logger output blocks when reporting responses generated by the AI provider.
      See Also:

    • EXIT_SPECIAL_PROMPT_COMMAND

      public static final String EXIT_SPECIAL_PROMPT_COMMAND
      Interactive-mode command that terminates processing immediately by throwing a ProcessTerminationException with exit code 0.

      Example interactive input: .

      See Also:

    • CONTINUE_SPECIAL_PROMPT_COMMAND

      public static final String CONTINUE_SPECIAL_PROMPT_COMMAND
      Interactive-mode command that accepts the current response and continues processing without sending another prompt to the provider.

      Example interactive input: >

      See Also:

    • NO_INTERACTIVE_SPECIAL_PROMPT_COMMAND

      private static final String NO_INTERACTIVE_SPECIAL_PROMPT_COMMAND
      Interactive-mode command that accepts the current response and continues processing in a non-interactive mode, running to completion without prompting the user again.

      Example interactive input: >>

      See Also:

    • model

      private String model
      The specific AI model identifier or provider label being utilized for operations.

    • instructions

      private String instructions
      Base instructions that set the persona, tone, and scope for the AI provider during execution.

    • defaultPrompt

      private String defaultPrompt
      Default fallback prompt used to process directories or projects when no specific file prompt is given.

    • interactive

      private boolean interactive
      Flag indicating whether interactive mode is active, allowing continuous prompt looping and execution.

    • functionTools

      private List<FunctionTools> functionTools
      Collection of user-configured function tools that are registered with and made available to the AI provider.

    • functionToolsLoader

      private static FunctionToolsLoader functionToolsLoader
      Loader utility responsible for discovering, instantiating, and applying functional tools to the AI provider.

  • Constructor Details

    • AIFileProcessor

      public AIFileProcessor(File rootDir, Configurator configurator, String genai)
      Creates a processor for the given project directory and AI provider identifier.
      Parameters:
      rootDir - the project root directory
      configurator - the application configuration
      genai - the AI provider or model identifier

  • Method Details

    • process

      public String process(ProjectLayout projectLayout, File file, String prompt)
      Processes the given file using the currently configured instructions.
      Parameters:
      projectLayout - the current project layout metadata
      file - the file to process
      prompt - the prompt to send to the AI provider
      Returns:
      the provider response, or null when no response is produced

    • process

      protected String process(ProjectLayout projectLayout, File file, String instructions, String... prompts)
      Processes a specified file within a project layout by configuring and invoking a Generative AI (GenAI) provider using a sequence of instructions and prompts.

      This method performs the following operations:

      • Establishes the thread context for the provided ProjectLayout.
      • Extracts input parameters from the provided array of prompts.
      • Resolves the GenAI model configuration (falling back to the default configured model if not explicitly overridden in prompt metadata via gw.model).
      • Instantiates the target Genai provider and registers enabled toolkits and custom function tools.
      • Constructs system instructions by combining default bundle instructions with any custom parameters passed to instructions.
      • Feeds file-specific contextual metadata and substituted prompts to the AI provider.
      • Executes the AI operation and returns the generated content.

      Supported Input Properties (extracted dynamically from the prompts' metadata):

      • gw.model (String) - Overrides the default model identifier used to initialize the GenAI provider. If not present, the method falls back to the default instance model.
      • enabledTools (String or List<?>) - Configures which toolkits or tools should be enabled for the AI provider. Defined via the constant ENABLED_TOOLS_PARAM_NAME.
      • Other YAML properties are retained as prompt configuration values and string values may be resolved through the active configurator before they are used. Properties not recognized by the processor are available for configuration substitution but do not otherwise alter processing.
      Parameters:
      projectLayout - the directory structure and metadata context of the active project
      file - the target file currently being processed
      instructions - additional custom system instructions to append to the default system instructions; can be null or blank
      prompts - a variable-length list or array of user prompt sequences to be evaluated and sent to the GenAI model
      Returns:
      the output string containing the model's response if processing was successful; null if prompts were empty or blank
      Throws:
      IllegalArgumentException - if the resolved GenAI model identifier is missing or no matching provider can be initialized

    • applyTools

      protected void applyTools(String instructions, String[] prompts, Genai provider, String[] tools)
      Registers the selected discovered tools and the tools explicitly added to this processor with the provider for the current request.
      Parameters:
      instructions - the resolved system instructions
      prompts - the resolved prompts
      provider - the provider that receives the tools
      tools - the selected tool names, or null for the default set

    • getEnabledTools

      private String[] getEnabledTools(Map<String,Object> inputProps, LayeredConfigurator conf)
      Resolves the tool names enabled for the current processing request.

      Prompt front matter takes precedence over the configured value. A string is split on whitespace, commas, and semicolons; a YAML list is converted to an array of item strings. Other value types are treated as though no explicit tool selection was supplied. When neither source provides a value, null is returned so the provider can use its default tool set.

      Parameters:
      inputProps - prompt parameters extracted from YAML front matter
      conf - layered configuration used as the fallback source
      Returns:
      enabled tool names, or null when no selection is configured

    • extractInputParams

      private String extractInputParams(String prompt, Map<String,Object> inputProps)
      Extracts supported YAML front-matter input parameters from the beginning of a prompt.

      A prompt may start with a YAML block delimited by ---. The block is removed from the returned prompt content and each YAML entry is merged into inputProps. String values are resolved with the processor configurator; non-string values, such as YAML lists, are preserved.

      Supported special input parameters include:

      • gw.model: model or provider identifier to use for this prompt.
      • enabledTools: a string or YAML list naming provider tools that should be enabled.

      Example: 

      
 ---
 gw.model: ${public.ai.model}
 enabledTools:
   - project-context
   - file-system
 ---
 Analyze this file.
      Parameters:
      prompt - the input prompt string which may contain a leading YAML configuration block
      inputProps - the map to populate with extracted and resolved parameters
      Returns:
      the stripped prompt content when a configuration block is present; otherwise, the original prompt string

    • removeFrontMatterData

      static String removeFrontMatterData(String prompt)

    • resolveInputParamValue

      private Object resolveInputParamValue(Object value)

    • getProcessInfo

      public String getProcessInfo(ProjectLayout projectLayout, File file)
      Generates a structured JSON string containing execution metadata about the file being processed and the current processing environment context.

      This method builds a map structure containing processing details and attempts to serialize it to a JSON format. The generated map holds the following properties:

      • "PROCESSED_FILE_REL_PATH" - The relative path of the processed file with respect to the project directory.
      • "PROCESS_MODE" - The current interaction mode, returning "INTERACTIVE" if the execution is interactive, otherwise "NOT-INTERACTIVE".

      Serialization Fallback: If Jackson's ObjectMapper fails to serialize the map to a standard JSON string, the method falls back to the default string representation of the map (via Object.toString()).

      Parameters:
      projectLayout - the layout configuration of the project, used to resolve the base project directory for path relative-ization; must not be null
      file - the file currently undergoing processing, used to determine its relative path; must not be null
      Returns:
      a string representation of the processing information map; ideally a valid JSON-formatted string, or a stringified map representation if serialization fails
      See Also:

    • perform

      private String perform(File file, Genai provider)

    • input

      protected String input()
      Obtains the next input from an interactive user session.

      The default implementation does not read from standard input because this processor may run without a supported interactive console. Subclasses can override this method to provide console or UI input.

      Returns:
      the next command or prompt, or null when input is unavailable

    • setProjectLayoutContext

      private void setProjectLayoutContext(ProjectLayout projectLayout)
      Extracts context metadata from the provided ProjectLayout and registers it in the project context registry.

      This method evaluates essential environment information (such as the operating system), project structure configurations (like name, IDs, and relative paths), and maps directory definitions (such as source files, tests, documentation, and sub-modules) into a centralized storage registry managed by ProjectContextFunctionTools.

      Directory collections are consolidated into formatted string information lines relative to the project directory before registration.

      Parameters:
      projectLayout - the ProjectLayout containing the current project structure, directories, and parent configurations; must not be null
      Throws:
      IllegalArgumentException - if an error occurs during JSON serialization or parsing of the layout information (wraps JsonProcessingException)
      See Also:

    • getDirInfoLine

      com.fasterxml.jackson.databind.node.ArrayNode getDirInfoLine(Collection<String> sources, File projectDir)
      Returns a JsonNode (ArrayNode) containing the names of directories from the given collection that exist within the specified project directory. Each directory name is wrapped in backticks.
      Parameters:
      sources - a collection of directory names (relative to projectDir) to check for existence
      projectDir - the base directory in which to check for the existence of each source directory
      Returns:
      a JsonNode (ArrayNode) of existing directory names, each wrapped in backticks (e.g., ["`src`", "`resources`"])

    • setInstructions

      public void setInstructions(String instructions)
      Sets the base instructions used for processing after normalizing line content and resolving supported references.
      Parameters:
      instructions - the raw instruction text

    • getInstructions

      public String getInstructions()
      Returns the current base instructions used for processing.
      Returns:
      the configured instruction text

    • parseLines

      public String parseLines(String data, File projectDir, Configurator conf)
      Normalizes multi-line input and resolves supported line references such as HTTP URLs and file: references.
      Parameters:
      data - the input text to parse
      projectDir - the project root used to resolve relative file references
      conf - the configurator used to substitute public properties
      Returns:
      the normalized text

    • tryToGetFromReference

      String tryToGetFromReference(String data, File projectDir, Configurator conf) throws IOException
      Resolves a single instruction line that may point to external content.

      If the instruction contains an external reference marker, it will be fetched and resolved. URLs starting with http:// or https:// are fetched remotely. URIs starting with file:// are resolved and loaded relative to the provided project directory.

      Parameters:
      data - the instruction line to inspect
      projectDir - the root directory of the project, used as the base context to resolve relative file:// references
      conf -
      Returns:
      the resolved content, or the original line when no reference is found, or null when the input is null
      Throws:
      IOException - if the referenced remote content or local file cannot be read

    • readFromHttpUrl

      static String readFromHttpUrl(String urlString) throws IOException
      Reads UTF-8 text content from the given HTTP or HTTPS URL.
      Parameters:
      urlString - the URL to read
      Returns:
      the response content as text
      Throws:
      IOException - if the URL cannot be read

    • readFromFilePath

      String readFromFilePath(String filePath, File projectDir) throws IOException
      Reads UTF-8 text content from the given file path.
      Parameters:
      filePath - the absolute or project-relative file path
      projectDir - the project root used to resolve a relative path
      Returns:
      the file content as text
      Throws:
      IOException - if the file cannot be opened or read

    • scanDocuments

      public void scanDocuments(File projectDir, String path) throws IOException
      Configures scanning based on the provided directory or path pattern and then starts scanning the project folder.

      The path argument may be specified as:

      • An absolute path — used as-is to scan a specific location.
      • A relative path — resolved against projectDir.
      • A glob pattern — e.g., "glob:**&#47;*.java", matched against files under projectDir.
      • A regex pattern — e.g., "regex:.*\\.java", matched against files under projectDir.
      If path equals the absolute path of projectDir, the entire project directory is scanned without applying any pattern matching.
      Parameters:
      projectDir - the project root directory; must not be null
      path - the directory, relative path, glob pattern, or regex pattern used to match files to scan; must not be blank
      Throws:
      IllegalArgumentException - if projectDir is null or path is blank
      IOException - if scanning fails

    • parsePath

      String parsePath(File projectDir, String path)
      Resolves the effective scan directory and converts it into a glob expression when required.
      Parameters:
      projectDir - the base project directory
      path - the configured scan directory
      Returns:
      the resolved path matcher expression

    • getDefaultPrompt

      public String getDefaultPrompt()
      Returns the default prompt used for folder processing.
      Returns:
      the default prompt

    • setDefaultPrompt

      public void setDefaultPrompt(String defaultPrompt)
      Sets the default prompt used for folder processing.
      Parameters:
      defaultPrompt - the default prompt text

    • processFolder

      public void processFolder(ProjectLayout projectLayout)
      Processes the project root folder using the configured default prompt.
      Overrides:
      processFolder in class AbstractFileProcessor
      Parameters:
      projectLayout - the current project layout metadata

    • getModel

      public String getModel()
      Returns the configured AI model or provider identifier.
      Returns:
      the model or provider identifier

    • setModel

      public void setModel(String genai)
      Sets the AI model or provider identifier.
      Parameters:
      genai - the model or provider identifier

    • setInteractive

      public void setInteractive(boolean interactive)
      Enables or disables interactive processing mode.
      Parameters:
      interactive - true to enable interactive mode; otherwise false

    • isInteractive

      public boolean isInteractive()
      Indicates whether interactive processing mode is enabled.
      Returns:
      true when interactive mode is enabled; otherwise false

    • addTool

      public void addTool(FunctionTools toolFunction)
      Adds a tool definition that will be exposed to the AI provider.
      Parameters:
      toolFunction - the tool definition to add