Toucan Connectors

Toucan Toco data connectors are plugins to the Toucan Toco platform. Their role is to return Pandas DataFrames from many different sources.

Each connector is dedicated to a single type of source (PostrgeSQL, Mongo, Salesforce, etc...) and is made of two classes:

• Connector which contains all the necessary information to use a data provider (e.g. hostname, auth method and details, etc...).
• DataSource which contains all the information to get a dataframe (query, path, etc...) using the Connector class above.

The Toucan Toco platform instantiates these classes using values provided by Toucan admin and app designers, it then uses the following methods to get data and metadata:

• Connector._retrieve_data returning an instance of pandas.DataFrame, method used to return data to a Toucan Toco end user
• Connector.get_slice returning an instance of DataSlice, method used to return data to a Toucan Toco application designer when building a query.
• Connector.get_status returning an instance of ConnectorStatus, method used to inform an admin or Toucan Toco application designer of the status of its connection to a third party data service. Is it reachable from our servers? Are the authentication details and method working? etc...

Installing for development

We use uv for packaging and development. Use the following command to install the project for development:

uv sync --all-extras

Dependencies

This project uses make and Python 3.8. Install the main dependencies :

pip install -e .

We are using the setuptools construct extra_requires to define each connector's dependencies separately. For example to install the MySQL connector dependencies:

pip install -e ".[mysql]"

There is a shortcut called all to install all the dependencies for all the connectors. I do not recommend that you use this as a contributor to this package, but if you do, use the section below to install the necessary system packages.

pip install -e ".[all]"

You may face issues when instally the repo locally due to dependencies. That's why a dev container is available to be used with visual studio. Refer to this doc to use it.

System packages

Some connectors dependencies require specific system packages. As each connector can define its dependencies separatly you do not need this until you want to use these specific connectors.

ODBC

On linux, you're going to need bindings for unixodbc to install pyodbc from the requirements, and to install that (using apt), just follow:

sudo apt-get update
sudo apt-get install unixodbc-dev

MSSSQL

To test and use mssql (and azure_mssql) you need to install the Microsoft ODBC driver for SQL Server for Linux or MacOS

PostgreSQL

On macOS, to test the postgres connector, you need to install postgresql by running for instance brew install postgres. You can then install the library with env LDFLAGS='-L/usr/local/lib -L/usr/local/opt/openssl/lib -L/usr/local/opt/readline/lib' pip install psycopg2

Other

You can find all connectors specific documentation here

Testing

We are using pytest and various packages of its ecosystem. To install the testing dependencies, run:

pip install -r requirements-testing.txt

As each connector is an independant plugin, its tests are written independently from the rest of the codebase. Run the tests for a specifc connector (http_api in this example) like this:

pytest tests/http_api

Note: running the tests above implies that you have installed the specific dependencies of the http_api connector (using the pip install -e .[http_api] command)

Our CI does run all the tests for all the connectors, like this:

pip install -e ".[all]"
make test

Some connectors are tested using mocks (cf. trello), others are tested by making calls to data providers (cf. elasticsearch) running on the system in docker containers. The required images are in the tests/docker-compose.yml file, they need to be pulled (cf. pytest --pull) to run the relevant tests.

Contributing

This is an open source repository under the BSD 3-Clause Licence. The Toucan Toco tech team are the maintainers of this repository, we welcome contributions.

At the moment the main use of this code is its integration into Toucan Toco commercially licenced software, as a result our dev and maintenance efforts applied here are mostly driven by Toucan Toco internal priorities.

This is the beginning of the README. Read the rest at the original. Read the full README ↗