Project Background

Integrating our ioTORQ data platform with industrial facilities often requires the deployment of specific hardware and software equipment to combine multiple on-site data sources and stream data to the cloud. In order to speed up this process, I build a lightweight Extract-Transform-Load desktop software using Python. The architecture of this ETL software is largely inspired by Apache NIFI. Data processors can be combined to query data from local process historian (SQL servers), transform the data to JSON and send it to our API. It relies on Asyncio for concurrent execution and runs the data processors in parallel.

The software includes a graphical user interface and configuration files so that it can be setup by non-developers.

Technologies Used