Creating a dataset from a remote source (URL, API, FTP)

As a source of data for a dataset, you can add a file from your computer, a URL (a remote server or API), or an FTP server (using the FTP, FTPS, or SFTP protocol).

The size limit for a file is 240 Mo. If your file is too big, you can compress it before uploading it.

For more information on compressed or uncompressed file formats, see Supported file formats.

You can connect the platform to:

• A remote server via its URL, to import files stored there

• A remote source that exposes data records over an API

Huwise supports HTTP and HTTPS URLs to files, for example "http://example.org/mydata.csv".

Huwise supports the following authenticated connection methods: login/password, API key, OAuth2 protocol (client_credentials grant type) and supplying info within an HTTP header.

If you are using an API that was provided to you, note that it must meet the following requirements:
- Be of the REST type and preferably in JSON, with a "GET" method
- It must be available via the internet
- It must not be paginated or incremental, i.e. results must be obtained in a single call
- It must be accessible by the Huwise platform via simple authentication (permanent API key, fixed HTTP header, etc.)

1. Go to Catalog > Assets and click on Create an asset and select Dataset

2. On the page that opens, select From a URL under "Retrieve a file"

3. Configure your HTTP connection

4. Using the preview of the first 20 records, configure the source

5. Configure the dataset's name and category

Huwise does not support implicit FTP as it is considered deprecated.

Huwise supports FTP, FTPS, or SFTP URLs to:

• Files, for example: "ftp://example.org/my_dir/mydata.csv"

• Folders, for example: "ftp://example.org/my_dir/"

1. Go to Catalog > Assets and click on Create an asset and select Dataset

1. On the page that opens, select From an FTP server under "Retrieve a file"

2. Configure your connection

SSH certificate authentication is available for SFTP servers.Regarding SSH certificate configuration:- paste the entire private key file in Private Key field- the passphrase is relative to the private key and is optional

3. Using the preview of the first 20 records, configure the source

4. Configure the dataset's name and category

After creating a dataset, an interface for that dataset opens. Only users with the right permissions, either "Create dataset" or "Edit dataset", can use this interface. There you can process the data, configure the datasets and their visualizations, and publish the datasets.

If you update your source file and republish, the entire file will be loaded. However, if the source is a folder and not a file, Huwise will take into account only the newly uploaded files within that folder.

If you delete a file from the FTP folder, you need to perform a small piece of upkeep. To the right of the resource, click on Clean cache before republishing, or else the change will not be reflected in the related dataset.

For more information about FTP folders, please read this section about sourcing multiple files stored on an FTP server.

You can create datasets based on data from remote services for which we have created dedicated connectors.

For the list of available connectors, see here. Some connectors are only available on-demand and may be subject to pricing.