How to create LLM tools from any Python SDK using langchain-autotools

Providing LLMs with access to our internal and external APIs and databases is one of the most promising ways of using LLMs in the business context. While chat-bot-applications like ChatGPT are certainly great and interesting, the productivity gains of LLMs just start there - they are much much higher if we find ways to integrate these models into our existing workflows or create new workflows for our daily businesses.

The concept for how to enable LLMs to interact with other technologies is already quite established - it's called Tools.

While the concept of tools is indeed very good, it's not that straightforward to design these tools in a way that our LLMs can work with them. And, at the end of the day, it's a lot of work to create them.

That's where today's blog topic comes into play: langchain-autotools. It's a community-maintained library integrating with the well-known Langchain SDK. The idea behind langchain-autotools is to make the creation of tools for LLMs as easy as possible.

What are LLM tools?

Tools in the context of LLMs, in its purest form, are just functions of code that are run after an LLM decides to use them.

The whole process is - in general - rather simple:

1. Create a code runtime environment, where you can run code. A python environment for example.

2. Create a function that for example interacts with an API or a database. The function can also have parameters, they will be provided by the LLM later on.

3. Describe, in natural language, what this function does. Also describe the parameters and the return value.

4. In your application, when you want an LLM to execute a specific task, send your task prompt the LLM, as well as all the tool (think, function) descriptions. The LLM will then decide whether it can fulfill your prompt without using a tool - or if it needs a tool.

5. If the LLM decides to use a tool, it will tell you which tool to use. You then simply call your function with the parameters provided by the LLM. In pseudo python code, this is just an if statement:

1## Note: This is example code, not real one
2def database_tool(query):
3    database_client = new Client()
4    result = database_client.query(query)
5    return result
6
7system_prompt = """You are a system to answer questions. You get a list of
8tools which you can use to answer the question.
9Either answer the users question directly, or answer with the tool to user.
10If you need a tool, answer with the following format. If required by the
11tool, make sure to provide the tool parameters.
12
13{"tool": "database_tool", "parameters": {"query": "select count(*) from table"}}
14
15Tools:
16
171. Database Tool: name: database_tool
18parameters:
19- query: The database PostgreSQL query to run
20"""
21
22response = LLM.prompt("How many users are in our database?")
23
24if json.loads(response)["tool"]:
25    tool = json.loads(response)["tool"]
26    if tool == "database_tool":
27        ## Call your tool function
28        result = database_tool(json.loads(response)["parameters"]["query"])

To summarize: A tool is nothing more then a function with a good description so that the LLM knows what it does and how to use it.

If you're dealing with a lot of external system to engage with, you might already see the problem: It's quite cumbersome to wrap all your external systems into functions and provide each function with a good description.

Especially when we talk about using SDKs (and many external systems nowadays provide SDKs), they already have quite extensive documentation. Why can't we use this documentation to create tools for our LLMs?

How to create LLM tools with langchain-autotools

Well, the answer is: We can. And that's what langchain-autotools is for. It combines the Langchain Agent concept with said SDK documentation - for enabling these agents to use any SDK as their tool.

The process of creating a tool now gets tremendously simplified:

1. Instantiate a client object of the SDK you want to use.

2. Create an autotool, with the client you just created.

3. Create an agent, using a predefined prompt and using this autotool.

4. Execute the prompt with your agent.

In pseudo - code this looks like this:

1database_client = new Client(secrets) # Your 'normal' SDK client creation
2client = {"client": database_client}
3
4autotool = AutoToolWrapper(client=client)
5
6llm = ChatOpenAI(model="gpt-4o", temperature=0)
7prompt = hub.pull("hwchase17/structured-chat-agent") # This pulls a predefined prompt from the langchain prompt hub
8agent = create_structured_chat_agent(llm, autotool.operations, prompt)
9agent_executor = AgentExecutor(
10    agent=agent,
11    tools=autotool.operations, # Contains the descriptions of the functions your SDK provides
12    verbose=True, # How much information you want to see in the logs
13    handle_parsing_errors=True,
14    max_iterations=5,
15)
16
17agent_executor.invoke(
18    {
19        "input": "How many users are in our database?",
20    }
21)

As you can see the tedious part of wrapping your SDK functions into well- described functions is gone.

To explain the parameters of the AgentExecutor:

• agent is the agent object you created, using an LLM as well as a prompt.
• prompt: You can supply your own prompt, or use a predefined prompt. The Langchain prompt hub is an excellent place to find predefined prompts - and Langchain makes it easy to use them, by simply calling hub.pull("author/prompt-name").
• tools: A list of tools to supply. autotool.operations contains a list of all available functions of your SDK client.
• handle_parsing_errors allows to define how the agent should react, when a tool can't be correctly invoked, as the LLM answered with an answer, which is not in the expected format - basically each time a runtime error after LLM response happens. By default, the execution will be aborted and an exception is raised. Setting this to true will send the error message along with the original task to the LLM again - and the execution is repeated. This is a surprisingly robust way of dealing with such errors. You can also provide your own error handling functions, read more about it in the Langchain documentation.
• max_iterations is the maximum number of iterations the agent should try to execute the task. One iteration is one LLM call. So if you want the LLM to fetch your database user count, the first call will most likely result in a tool call. The second iteration will be you presenting the result to the agent, to decide whether the task is finished. This task would therefore require 2 iterations.

Specifying the functions to expose to the LLM

You might already object and note, that SDKs typically provide a lot of functions. Meaning, we would send all of them to the LLM which would probably overload the LLMs context window or at least invoke unnecessary costs. Most of the time, we'll only need two or three functions anyhow.

Langchain-autotools provides a way to filter the functions one wants to expose to the LLM, by creating CrudControls. These CrudControls allow to define CRUD verbs - only functions which match these verbs will be exposed.

More specifically, one can define the following CRUD verbs:

• create
• read
• update
• delete

The CrudControls interface allows to define

Let's see an example. Imagine we want to only expose the query and queryMany functions and the drop_database function, we could define it like this:

1crud_controls = CrudControls(
2    read = True, # You need to enable each CRUD verb you want to use
3    read_list="query, queryMany", # Define multiple functions per CRUD operations using comma separation
4    delete = True,
5    delete_list="drop_database"
6)

Note that you specifically need to enable the CRUD verbs to use. Just providing a list of functions per verb is not enough.

The available parameters of CrudControls are:

• read: Enable the read verb
• read_list: Comma-separated list of function-names to expose to the LLM
• create: Enable the create verb
• create_list: Comma-separated list of function-names to expose to the LLM
• update: Enable the update verb
• update_list: Comma-separated list of function-names to expose to the LLM
• delete: Enable the delete verb
• delete_list: Comma-separated list of function-names to expose to the LLM

To use the created CrudControls, simply pass it to the AutoToolWrapper:

1autotool = AutoToolWrapper(client=client, crud_controls=crud_controls)

Example: How to create a tool to interact with Azure Blog Storage

Now that we know how to create tools using langchain-autotools, let's engage in a real world example. Let's say I have an Azure Blob Storage where I store some data required for some of my customer service requests.

We could create a small customer service LLM agent, which fetches documents from the Blob Storage if required and answers questions using these documents. In general, our agent therefore needs the following capabilities:

• Listing files in our blob storage
• Downloading files and extracting the content

We'll also use this as an example to demonstrate one of the limitations of the autotools library. But first things first, let's get started.

Lets start by installing the required libraries:

1pip install azure-storage-blob azure-identity langchain-autotools langchain==0.2.16 langchain_openai==0.1.25

Get the connection string from your Azure Blob Storage account

Azure Blob Storage Connection String

Load the required dependencies and initialize the Azure Blob Storage client:

1import os
2from azure.identity import DefaultAzureCredential
3from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient
4
5from langchain_autotools import AutoToolWrapper, CrudControls
6from langchain import hub
7from langchain_core.tools import tool
8
9from langchain.agents import AgentExecutor, create_structured_chat_agent
10from langchain_openai import ChatOpenAI
11
12os.environ["AZURE_STORAGE_CONNECTION_STRING"] = "<your connection string>"
13blob_service_client = BlobServiceClient.from_connection_string(os.environ["AZURE_STORAGE_CONNECTION_STRING"])
14container_client = blob_service_client.get_container_client("blogphdata")

Next, we define the autotools client and the CRUD controls. We can find the correct function names of our Azure blob storage SDK in the documentation.

1client = {"client": container_client}
2crud_controls = CrudControls(
3    read=True,
4    read_list="list_blobs,download_blobs",
5)

All we have to do is to initialize the autotools and Langchain as described in the chapter above:

1autotool = AutoToolWrapper(client=client, crud_controls=crud_controls)
2
3prompt = hub.pull("hwchase17/structured-chat-agent")
4
5
6llm = ChatOpenAI(model="gpt-4o", temperature=0)
7
8tools = autotool.operations + [download_blob]
9print(tools)
10agent = create_structured_chat_agent(llm, tools, prompt)
11agent_executor = AgentExecutor(agent=agent,
12                               tools=tools,
13                               verbose=True,
14                               handle_parsing_errors=True,
15                               max_iterations=10)
16
17agent_executor.invoke(
18    {
19        "input": "Use your tools to find the results in one of the blobs.",
20        "input": "How much is the price of the product 'Kopfhörer'?"
21    }
22)

But what is that? Most probably your agent didn't respond with the correct answer, even if the answer was provided in one of the blobs. Let's look at the output of the agent to find out why.

1{
2  "action": "download_blob",
3  "action_input": {
4    "blob": "test.txt"
5  }
6}
7"<azure.storage.blob._download.StorageStreamDownloader object at 0x7f99640ae350>"
8
9> Finished chain.
10
11{'input': "How much is the price of the product 'Kopfhörer'? Use your tools to find the answer in one of the blobs",
12 'output': 'Agent stopped due to iteration limit or time limit.'}

Well, it's pretty clear: The download_blob function returns a StorageStreamDownloader, which in itself is not the content of the blob itself.

One would need to call download_blob('blobname').readall() to get the real text content. However, this is not possible as of now with langchain-autotools. One can only call top-level functions of the SDK. To circumvent this, we can manually create an describe our own function, as follows:

1@tool
2def download_blob(blob_name):
3    """Download a blog from the container and return the text content"""
4
5    data = container_client.download_blob(blob_name).readall()
6    return data

Note the @tool decorator, which makes our function a Langchain tool.

We can combine our automatically created autotool-functions with our own function as follows:

1crud_controls = CrudControls(
2    read=True,
3    read_list="list_blobs", # remove the download_blob function, as we'll create our own
4)
5
6tools = autotool.operations + [download_blob]
7agent = create_structured_chat_agent(llm, tools, prompt)
8agent_executor = AgentExecutor(agent=agent,
9                               tools=tools,
10                               verbose=True,
11                               handle_parsing_errors=True,
12                               max_iterations=10)

If we run our agent_executor once more, we'll get the correct answer, if it is provided in one of the blobs.

Further reading

• Extract structured data for LLMs from any website
• Chat with your confluence using LLMs
• How to fine-tune your very own LLM

Interested in how to train your very own Large Language Model?

We prepared a well-researched guide for how to use the latest advancements in Open Source technology to fine-tune your own LLM. This has many advantages like:

• Cost control
• Data privacy
• Excellent performance - adjusted specifically for your intended use