kani (カニ)

kani (カニ) is a lightweight and highly hackable framework for chat-based language models with tool usage/function calling.

Compared to other LM frameworks, kani is less opinionated and offers more fine-grained customizability over the parts of the control flow that matter, making it the perfect choice for NLP researchers, hobbyists, and developers alike.

kani comes with support for the following models out of the box, with a model-agnostic framework to add support for many more:

• OpenAI Models (pip install "kani[openai]")
• Anthropic Models (pip install "kani[anthropic]")
• Google AI Models (pip install "kani[google]")
• Hugging Face transformers (pip install "kani[huggingface]")
• llama.cpp (pip install "kani[cpp]")
• vLLM (pip install "kani[vllm]")
• and more with community extensions!

Check out the Model Zoo for code examples of loading popular models in Kani!

Interested in contributing? Check out our guide.

Read the docs on ReadTheDocs!

Read our paper on arXiv!

Installation

kani requires Python 3.10 or above. To install model-specific dependencies, kani uses various extras (brackets after the library name in pip install). To determine which extra(s) to install, see the model table, or use the [all] extra to install everything.

# for OpenAI models
$ pip install "kani[openai]"
# for Hugging Face models
$ pip install "kani[huggingface]" torch
# for multimodal inputs
$ pip install "kani[multimodal]"
# or install everything:
$ pip install "kani[all]"

For the most up-to-date changes and new models, you can also install the development version from Git's main branch:

$ pip install "kani[all] @ git+https://github.com/zhudotexe/kani.git@main"

Quickstart

kani requires Python 3.10 or above.

First, install the library. In this quickstart, we'll use the OpenAI engine, though kani is model-agnostic.

$ pip install "kani[openai]"

Then, let's use kani to create a simple chatbot using ChatGPT as a backend.

# import the library
import asyncio
from kani import Kani, chat_in_terminal
from kani.engines.openai import OpenAIEngine

# Replace this with your OpenAI API key: https://platform.openai.com/account/api-keys
api_key = "sk-..."

# kani uses an Engine to interact with the language model. You can specify other model 
# parameters here, like temperature=0.7.
engine = OpenAIEngine(api_key, model="gpt-5-nano")

# The kani manages the chat state, prompting, and function calling. Here, we only give 
# it the engine to call ChatGPT, but you can specify other parameters like 
# system_prompt="You are..." here.
ai = Kani(engine)

# kani comes with a utility to interact with a kani through your terminal...
chat_in_terminal(ai)


# or you can use kani programmatically in an async function!
async def main():
    resp = await ai.chat_round("What is the airspeed velocity of an unladen swallow?")
    print(resp.text)


asyncio.run(main())

kani makes the time to set up a working chat model short, while offering the programmer deep customizability over every prompt, function call, and even the underlying language model.

Function Calling

Function calling gives language models the ability to choose when to call a function you provide based off its documentation.

[!NOTE] Looking for MCP support? Kani supports local and remote MCP servers, too. Check out the MCP docs at https://kani.readthedocs.io/en/latest/function_calling.html#mcp-tools.

With kani, you can write functions in Python and expose them to the model with just one line of code: the @ai_function decorator.

# import the library
import asyncio
from typing import Annotated
from kani import AIParam, Kani, ai_function, chat_in_terminal, ChatRole
from kani.engines.openai import OpenAIEngine

# set up the engine as above
api_key = "sk-..."
engine = OpenAIEngine(api_key, model="gpt-4o-mini")


# subclass Kani to add AI functions
class MyKani(Kani):
    # Adding the annotation to a method exposes it to the AI
    @ai_function()
    def get_weather(
        self,
        # and you can provide extra documentation about specific parameters
        location: Annotated[str, AIParam(desc="The city and state, e.g. San Francisco, CA")],
    ):
        """Get the current weather in a given location."""
        # In this example, we mock the return, but you could call a real weather API
        return f"Weather in {location}: Sunny, 72 degrees fahrenheit."


ai = MyKani(engine)

# the terminal utility allows you to test function calls...
chat_in_terminal(ai)


# and you can track multiple rounds programmatically.
async def main():
    async for msg in ai.full_round("What's the weather in Tokyo?"):
        print(msg.role, msg.text)


asyncio.run(main())

kani guarantees that function calls are valid by the time they reach your methods while allowing you to focus on writing code. For more information, check out the function calling docs.

Streaming

kani supports streaming responses from the underlying language model token-by-token, even in the presence of function calls. Streaming is designed to be a drop-in superset of the chat_round and full_round methods, allowing you to gradually refactor your code without ever leaving it in a broken state.

This is the beginning of the README. Read the rest at the original. Read the full README ↗