Package org.machanism.machai.gw.tools

Class WebFunctionTools

java.lang.Object
org.machanism.machai.gw.tools.WebFunctionTools
All Implemented Interfaces:
FunctionTools
public class WebFunctionTools extends Object implements FunctionTools
Provides host-side HTTP retrieval tools for a Genai provider.

This tool set exposes two main functions:

  • get_web_content – Fetches web page content over HTTP(S) via GET, optionally returning plain text or content selected via a CSS selector.
  • call_rest_api – Executes a generic REST call using an arbitrary HTTP method with optional headers and request body.

Header variable placeholders

Header values may include placeholders in the form ${propertyName}. When a Configurator is provided, those placeholders are resolved at runtime.

Authentication

HTTP Basic authentication is supported via the URL userInfo component (e.g., https://user:password@host/path), which is converted into an Authorization: Basic ... header. You can also specify an explicit Authorization header.

Outbound network policy (allow/deny lists) is intentionally left to the host application.

Usage Example

 WebFunctionTools tools = new WebFunctionTools();
 String html = tools.getWebContent("https://example.com", null, 5000, "UTF-8", false, "", projectDir, configurator);
 String apiResult = tools.callRestApi("https://api.example.com", "POST", headers, body, 5000, "UTF-8", projectDir,
                configurator);
Author:
Viktor Tovstyi

  • Field Details

    • DEFAULT_CHARSET

      private static final String DEFAULT_CHARSET
      Default response character set used by the HTTP tools.
      See Also:

    • REQUEST_ID_RANDOM

      private static final SecureRandom REQUEST_ID_RANDOM
      Source of correlation identifiers used in request log messages.

    • logger

      private static final org.slf4j.Logger logger
      Logger for web fetch tool execution and diagnostics.

  • Constructor Details

    • WebFunctionTools

      public WebFunctionTools()

  • Method Details

    • getWebContent

      public String getWebContent(String url, Map<String,String> headers, int timeout, String charsetName, boolean textOnly, String selector, File projectDir, Configurator configurator)
      Functional AI tool that fetches a web page or project-scoped file URL. The response may be filtered by a CSS selector and rendered as plain text.
      Parameters:
      url - URL to fetch; may contain Basic-authentication user info
      headers - optional request headers
      timeout - connection and read timeout in milliseconds, or zero for the connection default
      charsetName - character set used to decode the response
      textOnly - whether to strip HTML markup from the response
      selector - optional CSS selector used to select response elements
      projectDir - project root used to resolve file URLs
      configurator - configuration used to substitute URL and header values
      Returns:
      fetched, optionally selected and rendered content, or an error message

    • readFileUriContent

      private String readFileUriContent(File projectDir, String charsetName, URI uri)

    • fetchHttpContent

      private String fetchHttpContent(String requestId, Map<String,String> headers, int timeout, String charsetName, URI uri, Configurator config) throws IOException
      Fetches HTTP content for a parsed URI.
      Parameters:
      requestId - request correlation identifier
      headers - optional request headers
      timeout - timeout in milliseconds
      charsetName - response character set
      uri - target URI
      config - configuration used for header substitution
      Returns:
      response content
      Throws:
      IOException - if the connection or response cannot be read

    • readFileContent

      private String readFileContent(File file, String charsetName)
      Reads a local file using the requested character set.
      Parameters:
      file - file to read
      charsetName - character set used to decode the file
      Returns:
      file content or a not-found message

    • applySelectorIfPresent

      String applySelectorIfPresent(String selector, String response)
      Applies a CSS selector to the response HTML if one was provided.
      Parameters:
      selector - CSS selector (may be blank)
      response - full response content
      Returns:
      selected HTML content (joined with newlines) or the original response if selector is blank

    • renderTextOnlyIfRequested

      private String renderTextOnlyIfRequested(boolean textOnly, String response)
      Converts the response to plain text when requested.
      Parameters:
      textOnly - whether to render text only
      response - response content (typically HTML)
      Returns:
      rendered text content if textOnly is true; otherwise the original response

    • getConnection

      HttpURLConnection getConnection(URI uri, Map<String,String> headers, Configurator config) throws IOException
      Creates and configures an HttpURLConnection.

      If the URI contains userInfo, it is removed from the request URI and used to set an HTTP Basic Authorization header.

      Parameters:
      uri - URI to connect to
      headers - optional headers
      config - configuration used to resolve header placeholders
      Returns:
      connection
      Throws:
      IOException - if opening a connection fails

    • getWebPage

      String getWebPage(HttpURLConnection connection, int timeout, String charsetName) throws IOException
      Performs the HTTP GET request and returns the response content.
      Parameters:
      connection - open connection
      timeout - timeout in milliseconds
      charsetName - charset used to decode the response
      Returns:
      response content including an initial status line
      Throws:
      IOException - if the request cannot be executed

    • callRestApi

      public String callRestApi(String url, String method, Map<String,String> headers, String body, int timeout, String charsetName, File projectDir, Configurator configurator) throws IOException
      Functional AI tool that executes a REST API call to the specified URL using the given HTTP method.

      Supports userInfo format in the URL for basic authentication, custom headers, request body, timeout, and charset. Handles HTTP methods such as GET, POST, PUT, PATCH, DELETE, etc. If the URL contains user credentials (e.g., https://user:password@host/path), they are used for HTTP Basic authentication. Header values may include property placeholders resolved via the provided Configurator.

      The response includes an initial status line (e.g., HTTP 200 OK) followed by the response body.

      Parameters:
      url - The URL of the REST endpoint. Supports userInfo format (e.g., https://user:password@host/path) for basic authentication.
      method - The HTTP method to use (GET, POST, PUT, PATCH, DELETE, etc.). Default is GET.
      headers - Specifies HTTP header properties. If null, no additional headers are sent.
      body - The request body to send (for POST, PUT, PATCH, etc.).
      timeout - The maximum time in milliseconds to wait for the HTTP response. If not specified, a default timeout will be used.
      charsetName - The name of the character set to use when decoding the response content. Default: UTF-8.
      projectDir - The project directory context for file-based URLs.
      configurator - The configuration object for property resolution and header placeholder substitution.
      Returns:
      The REST API response as a string, including the status line and response body, or an error message if the call fails.
      Throws:
      IOException - if the URL connection cannot be opened or configured

    • parseResult

      private String parseResult(String requestId, String charsetName, HttpURLConnection connection, int responseCode, StringBuilder response) throws IOException
      Reads the response stream and returns the full response text.
      Parameters:
      requestId - correlation id used for logs
      charsetName - response decoding charset
      connection - open connection
      responseCode - HTTP response code
      response - builder already containing the status line
      Returns:
      response text
      Throws:
      IOException - if reading the response fails

    • getConnection

      private HttpURLConnection getConnection(String requestId, String url, String charsetName, String method, int timeout, Map<String,String> headers, String body, Configurator config) throws IOException
      Throws:
      IOException

    • fillHeader

      void fillHeader(Map<String,String> headers, HttpURLConnection connection, Configurator configurator)
      Applies HTTP headers to the given connection, resolving any property placeholders.

      Each header entry is set as a request property on the HttpURLConnection. Header values may include placeholders in the form ${propertyName}, which are resolved using the provided Configurator.

      Parameters:
      headers - Map of header names to values. If null, no headers are applied.
      connection - The HttpURLConnection to configure.
      configurator - The Configurator used to resolve property placeholders in header values.