Package org.machanism.machai.gw.tools

Class WebFunctionTools

java.lang.Object
org.machanism.machai.gw.tools.WebFunctionTools
All Implemented Interfaces:
FunctionTools
public class WebFunctionTools extends Object implements FunctionTools
Provides host-side HTTP retrieval tools for a Genai provider.

This tool set exposes two main functions:

  • get_web_content – Fetches web page content over HTTP(S) via GET, optionally returning plain text or content selected via a CSS selector.
  • call_rest_api – Executes a generic REST call using an arbitrary HTTP method with optional headers and request body.

Header variable placeholders

Header values may include placeholders in the form ${propertyName}. When a Configurator is provided, those placeholders are resolved at runtime.

Authentication

HTTP Basic authentication is supported via the URL userInfo component (e.g., https://user:password@host/path), which is converted into an Authorization: Basic ... header. You can also specify an explicit Authorization header.

Outbound network policy (allow/deny lists) is intentionally left to the host application.

Usage Example


 WebFunctionTools tools = new WebFunctionTools();
 String html = tools.getWebContent("https://example.com", null, 5000, "UTF-8", false, "", projectDir, configurator);
 String apiResult = tools.callRestApi("https://api.example.com", "POST", headers, body, 5000, "UTF-8", projectDir,
 		configurator);
Author:
Viktor Tovstyi

  • Constructor Details

    • WebFunctionTools

      public WebFunctionTools()

  • Method Details

    • getWebContent

      public String getWebContent(String url, Map<String,String> headers, int timeout, String charsetName, boolean textOnly, String selector, File projectDir, Configurator configurator)
      Fetches the content of a web page using an HTTP GET request.

      Supports userInfo format in the URL for basic authentication, custom headers, timeout, charset, plain text extraction, and CSS selector filtering. If the URL uses the file scheme, content is read from the local file system.

      If textOnly is true, the returned content is stripped of HTML tags and rendered as plain text. If selector is provided, only the content matching the specified CSS selector is returned. If both selector and textOnly are set, only the text of the selected elements is returned.

      Header values may include property placeholders resolved via the provided Configurator. HTTP Basic authentication is supported via userInfo in the URL (e.g., https://user:password@host/path).

      Parameters:
      url - The URL of the web page to fetch. Supports userInfo format (e.g., https://user:password@host/path) for basic authentication.
      headers - Specifies HTTP header properties. If null, no additional headers are sent.
      timeout - The maximum time in milliseconds to wait for the HTTP response. If not specified, a default timeout will be used.
      charsetName - The name of the character set to use when decoding the response content. Default: UTF-8.
      textOnly - If true, only the plain text content of the web page is returned (HTML tags are stripped). If false or not specified, the full HTML content is returned.
      selector - If provided, extracts and returns only the content matching the specified CSS selector. If textOnly is also true, returns only the text of the selected elements; otherwise, returns their HTML.
      projectDir - The project directory context for file-based URLs.
      configurator - The configuration object for property resolution and header placeholder substitution.
      Returns:
      The fetched web content as a string, or an error message if the fetch fails.

    • callRestApi

      public String callRestApi(String url, String method, Map<String,String> headers, String body, int timeout, String charsetName, File projectDir, Configurator configurator) throws IOException
      Executes a REST API call to the specified URL using the given HTTP method.

      Supports userInfo format in the URL for basic authentication, custom headers, request body, timeout, and charset. Handles HTTP methods such as GET, POST, PUT, PATCH, DELETE, etc. If the URL contains user credentials (e.g., https://user:password@host/path), they are used for HTTP Basic authentication. Header values may include property placeholders resolved via the provided Configurator.

      The response includes an initial status line (e.g., HTTP 200 OK) followed by the response body.

      Parameters:
      url - The URL of the REST endpoint. Supports userInfo format (e.g., https://user:password@host/path) for basic authentication.
      method - The HTTP method to use (GET, POST, PUT, PATCH, DELETE, etc.). Default is GET.
      headers - Specifies HTTP header properties. If null, no additional headers are sent.
      body - The request body to send (for POST, PUT, PATCH, etc.).
      timeout - The maximum time in milliseconds to wait for the HTTP response. If not specified, a default timeout will be used.
      charsetName - The name of the character set to use when decoding the response content. Default: UTF-8.
      projectDir - The project directory context for file-based URLs.
      configurator - The configuration object for property resolution and header placeholder substitution.
      Returns:
      The REST API response as a string, including the status line and response body, or an error message if the call fails.
      Throws:
      IOException