Package org.machanism.machai.gw.tools

Class WebFunctionTools

java.lang.Object

org.machanism.machai.gw.tools.WebFunctionTools

All Implemented Interfaces:

FunctionTools

public class WebFunctionTools
extends Object
implements FunctionTools

Provides host-side HTTP retrieval tools for a ProcessProvider provider.

This tool set exposes two main functions:

• get_web_content – Fetches web page content over HTTP(S) via GET, optionally returning plain text or content selected via a CSS selector.
• call_rest_api – Executes a generic REST call using an arbitrary HTTP method with optional headers and request body.

Header variable placeholders

Header values may include placeholders in the form ${propertyName}. When a Configurator is provided, those placeholders are resolved at runtime.

Authentication

HTTP Basic authentication is supported via the URL userInfo component (e.g., https://user:password@host/path), which is converted into an Authorization: Basic ... header. You can also specify an explicit Authorization header.

Outbound network policy (allow/deny lists) is intentionally left to the host application.

Usage Example

 WebFunctionTools tools = new WebFunctionTools();
 String html = tools.getWebContent("https://example.com", null, 5000, "UTF-8", false, "", projectDir, configurator);
 String apiResult = tools.callRestApi("https://api.example.com", "POST", headers, body, 5000, "UTF-8", projectDir,
                configurator);
 

Author:

Viktor Tovstyi

Field Summary

Fields

Modifier and Type	Field	Description
private static final String	DEFAULT_CHARSET	Default response character set used by the HTTP tools.
private static final org.slf4j.Logger	logger	Logger for web fetch tool execution and diagnostics.
private static final SecureRandom	RANDOM	Source of correlation identifiers used in request log messages.

Constructor Summary

Constructors

Constructor	Description
WebFunctionTools()

Method Summary

Modifier and Type	Method	Description
(package private) String	applySelectorIfPresent(String selector, String response)	Applies a CSS selector to the response HTML if one was provided.
String	callRestApi(String url, String method, Map<String,String> headers, String body, int timeout, String charsetName, File projectDir, Configurator configurator)	Functional AI tool that executes a REST API call to the specified URL using the given HTTP method.
private String	fetchHttpContent(String requestId, Map<String,String> headers, int timeout, String charsetName, URI uri, Configurator config)	Fetches HTTP content for a parsed URI.
(package private) void	fillHeader(Map<String,String> headers, HttpURLConnection connection, Configurator configurator)	Applies HTTP headers to the given connection, resolving any property placeholders.
private HttpURLConnection	getConnection(String requestId, String url, String charsetName, String method, int timeout, Map<String,String> headers, String body, Configurator config)	Creates and configures a connection for a REST request, including optional request-body transmission.
(package private) HttpURLConnection	getConnection(URI uri, Map<String,String> headers, Configurator config)	Creates and configures an HttpURLConnection.
String	getWebContent(String url, Map<String,String> headers, int timeout, String charsetName, boolean textOnly, String selector, File projectDir, Configurator configurator)	Functional AI tool that fetches a web page or project-scoped file URL.
(package private) String	getWebPage(HttpURLConnection connection, int timeout, String charsetName)	Performs the HTTP GET request and returns the response content.
private String	parseResult(String requestId, String charsetName, HttpURLConnection connection, int responseCode, StringBuilder response)	Reads the response stream and returns the full response text.
private String	readFileContent(File file, String charsetName)	Reads a local file using the requested character set.
private String	readFileUriContent(File projectDir, String charsetName, String uri)	Resolves a project-scoped file: URI and reads the referenced file.
private String	renderTextOnlyIfRequested(boolean textOnly, String response)	Converts the response to plain text when requested.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

DEFAULT_CHARSET

private static final String DEFAULT_CHARSET

Default response character set used by the HTTP tools.

See Also:

• Constant Field Values

RANDOM

private static final SecureRandom RANDOM

Source of correlation identifiers used in request log messages.

logger

private static final org.slf4j.Logger logger

Logger for web fetch tool execution and diagnostics.

Constructor Details

WebFunctionTools

public WebFunctionTools()

Method Details

getWebContent

public String getWebContent(String url,
 Map<String,String> headers,
 int timeout,
 String charsetName,
 boolean textOnly,
 String selector,
 File projectDir,
 Configurator configurator)
                     throws IOException

Functional AI tool that fetches a web page or project-scoped file URL. The response may be filtered by a CSS selector and rendered as plain text.

Parameters:

url - URL to fetch; may contain Basic-authentication user info, or a file:// scheme with a relative path resolved against projectDir

headers - optional request headers

timeout - connection and read timeout in milliseconds, or zero for the connection default

charsetName - character set used to decode the response

textOnly - whether to strip HTML markup from the response

selector - optional CSS selector used to select response elements

projectDir - project root used to resolve file URLs

configurator - configuration used to substitute URL and header values

Returns:

fetched content, optionally restricted to matching elements and rendered as plain text

Throws:

IllegalArgumentException - if url is not a valid URI or charsetName does not name a supported charset

IOException - if the target file cannot be read or the HTTP request cannot be completed

readFileUriContent

private String readFileUriContent(File projectDir,
 String charsetName,
 String uri)
                           throws IOException

Resolves a project-scoped file: URI and reads the referenced file.

Parameters:

projectDir - project root used for relative file paths

charsetName - character set used to decode the file

uri - textual file URI to resolve

Returns:

content of the resolved file

Throws:

IllegalArgumentException - if charsetName does not identify a supported character set

IOException - if the resolved file cannot be read

fetchHttpContent

private String fetchHttpContent(String requestId,
 Map<String,String> headers,
 int timeout,
 String charsetName,
 URI uri,
 Configurator config)
                         throws IOException

Fetches HTTP content for a parsed URI.

Parameters:

requestId - request correlation identifier

headers - optional request headers

timeout - timeout in milliseconds

charsetName - response character set

uri - target URI

config - configuration used for header substitution

Returns:

response content

Throws:

IllegalArgumentException - if charsetName does not identify a supported character set

IOException - if the request cannot be configured, connected, or read

readFileContent

private String readFileContent(File file,
 String charsetName)
                        throws IOException

Reads a local file using the requested character set.

Parameters:

file - file to read

charsetName - character set used to decode the file

Returns:

decoded file content

Throws:

IllegalArgumentException - if charsetName does not identify a supported character set

IOException - if the file cannot be opened or read

applySelectorIfPresent

String applySelectorIfPresent(String selector,
 String response)

Applies a CSS selector to the response HTML if one was provided.

Parameters:

selector - CSS selector (may be blank)

response - full response content

Returns:

selected HTML content, joined with line separators, or the original response if selector is blank

Throws:

IllegalArgumentException - if selector is not valid CSS syntax

renderTextOnlyIfRequested

private String renderTextOnlyIfRequested(boolean textOnly,
 String response)

Converts the response to plain text when requested.

Parameters:

textOnly - whether to render text only

response - response content (typically HTML)

Returns:

rendered text content if textOnly is true; otherwise the original response

Throws:

IllegalArgumentException - if the response cannot be parsed as HTML

getConnection

HttpURLConnection getConnection(URI uri,
 Map<String,String> headers,
 Configurator config)
                         throws IOException

Creates and configures an HttpURLConnection.

If the URI contains userInfo, it is removed from the request URI and used to set an HTTP Basic Authorization header.

Parameters:

uri - URI to connect to

headers - optional headers

config - configuration used to resolve header placeholders

Returns:

connection

Throws:

IllegalArgumentException - if uri cannot be converted to a URL

IOException - if opening a connection fails

getWebPage

String getWebPage(HttpURLConnection connection,
 int timeout,
 String charsetName)
           throws IOException

Performs the HTTP GET request and returns the response content.

Parameters:

connection - open connection

timeout - timeout in milliseconds

charsetName - charset used to decode the response

Returns:

response content including an initial status line

Throws:

IllegalArgumentException - if charsetName does not identify a supported character set

IOException - if the request cannot be executed

callRestApi

public String callRestApi(String url,
 String method,
 Map<String,String> headers,
 String body,
 int timeout,
 String charsetName,
 File projectDir,
 Configurator configurator)
                   throws IOException

Functional AI tool that executes a REST API call to the specified URL using the given HTTP method.

Supports userInfo format in the URL for basic authentication, custom headers, request body, timeout, and charset. Handles HTTP methods such as GET, POST, PUT, PATCH, DELETE, etc. If the URL contains user credentials (e.g., https://user:password@host/path), they are used for HTTP Basic authentication. Header values may include property placeholders resolved via the provided Configurator.

The response includes an initial status line (e.g., HTTP 200 OK) followed by the response body.

Parameters:

url - The URL of the REST endpoint. Supports userInfo format (e.g., https://user:password@host/path) for basic authentication.

method - The HTTP method to use (GET, POST, PUT, PATCH, DELETE, etc.). Default is GET.

headers - Specifies HTTP header properties. If null, no additional headers are sent.

body - The request body to send (for POST, PUT, PATCH, etc.).

timeout - The maximum time in milliseconds to wait for the HTTP response. If not specified, a default timeout will be used.

charsetName - The name of the character set to use when decoding the response content. Default: UTF-8.

projectDir - The project directory context for file-based URLs.

configurator - The configuration object for property resolution and header placeholder substitution.

Returns:

the REST API response, including the status line and response body; when no response stream is available, a response-code summary

Throws:

IllegalArgumentException - if url is not a valid URI, method is invalid, or charsetName does not name a supported charset

IOException - if the URL connection cannot be opened, configured, or its response cannot be read

parseResult

private String parseResult(String requestId,
 String charsetName,
 HttpURLConnection connection,
 int responseCode,
 StringBuilder response)
                    throws IOException

Reads the response stream and returns the full response text.

Parameters:

requestId - correlation id used for logs

charsetName - response decoding charset

connection - open connection

responseCode - HTTP response code

response - builder already containing the status line

Returns:

response text

Throws:

IllegalArgumentException - if charsetName does not identify a supported character set

IOException - if reading the response fails

getConnection

private HttpURLConnection getConnection(String requestId,
 String url,
 String charsetName,
 String method,
 int timeout,
 Map<String,String> headers,
 String body,
 Configurator config)
                                 throws IOException

Creates and configures a connection for a REST request, including optional request-body transmission.

Parameters:

requestId - request correlation identifier used for logging

url - endpoint URL

charsetName - character set used to encode the request body

method - HTTP method

timeout - connection and read timeout in milliseconds, or zero for the connection default

headers - optional request headers

body - optional request body

config - configuration used for header substitution

Returns:

configured HTTP connection

Throws:

IllegalArgumentException - if the URL, HTTP method, or character set is invalid

IOException - if the connection cannot be opened or the request body cannot be written

fillHeader

void fillHeader(Map<String,String> headers,
 HttpURLConnection connection,
 Configurator configurator)

Applies HTTP headers to the given connection, resolving any property placeholders.

Each header entry is set as a request property on the HttpURLConnection. Header values may include placeholders in the form ${propertyName}, which are resolved using the provided Configurator.

Parameters:

headers - Map of header names to values. If null, no headers are applied.

connection - The HttpURLConnection to configure.

configurator - The Configurator used to resolve property placeholders in header values.