Data Engineering

• One infrastructure plan for 14 departments

  Designed a comprehensive infrastructure framework for a research institute impacting 14 departments, featuring flexible modules, unified data pipelines, and structured support strategies for long‑term adoption.

• A month of meter readings in 180 milliseconds

  Engineered hourly electricity consumption aggregation pipeline in Python / SQL / Bash + Jq, achieving 180ms for 30‑day datasets across heterogeneous JSONL sources.

• Made a geographical data pipeline 50x faster

  Accelerated geographical data pipeline performance by 50x by improving SQL programming and data modeling across PostgreSQL, MS SQL, and Google Cloud BigQuery.

• Improved GIS map performance 10x on PostgreSQL

  Improved geographical map application performance by 10x through strategic database transition from MSSQL to PostgreSQL, including the data cutover, optimizing processing and data security.

• Resolved 1,000 issues in geographical data

  Resolved 1,000 issues in geographical data and time‑series data, using GDAL, ArcGIS, PostGIS, Mapbox, QGIS, SQL (PL/pgSQL, Transact‑SQL), Bash, ensuring high‑quality big data processing.

• Designed and ran 6 ETL/ELT pipelines

  Designed, implemented, and administered 6 ETL/ELT pipelines, utilizing Google BigQuery, MSSQL, PostgreSQL, Shell scripting, PL/pgSQL, and Transact‑SQL, integrating data for efficient Python API processing.

• Delivered 8 Power BI projects with manuals

  Delivered 8 Power BI projects with comprehensive manuals, integrating Microsoft Power BI tools with NodeJS API and Python FastAPI for effective data analytics and visualization.

• Ran 100 PostgreSQL, MS SQL, BigQuery warehouses

  Architected, created, and managed 100 PostgreSQL, MS SQL, and Google BigQuery data warehouse databases with primarily GIS and time‑series data, optimizing performance and scalability.

• Accelerated PostgreSQL performance by 10x

  Accelerated PostgreSQL performance by 10x via strategic indexing, partitioning, and query optimization, enhancing database efficiency for user, tenant, geospatial, and time‑series electrical data.

• Automated GIS SaaS deployment with GitHub Actions

  Automated GIS SaaS application deployment, data processing, and reporting system using GitHub Actions CI/CD, Python, Bash, and SQL.

• Automated delivery of 20 GIS data pipelines

  Automated delivery of 20 GIS data pipelines and app data ETL processes, streamlining infrastructure automation and reporting.

• Optimized forecasting and investment strategies

  Optimized forecasting and investment strategies for 11 electricity grid operators, driving operational efficiency through data‑driven GIS solutions.

• Led 30+ GIS projects from build to support

  Led the development, deployment, and support of over 30 GIS projects, demonstrating expertise in PostgreSQL, Bash, Python, JavaScript, GDAL, ArcGIS, PostGIS, and Mapbox technologies.

• Saved 4,000 hours with GitHub and GitLab CI/CD

  Streamlined data analysis and software development processes, saving 4,000 hours by introducing GitHub, GitLab, Bash, and Python CI/CD practices.

• Developed a Data Analytics reporting system

  Developed a Data Analytics reporting system, increasing quarterly software revenue by 400% through Python‑based PDF reports.

• Automated data processing in Python and PL/pgSQL

  Automated data processing tasks using Shell scripting, PL/pgSQL, Python, and Transact‑SQL, increasing productivity and efficiency.

• Engineered 600 PL/pgSQL ETL/ELT pipelines

  Engineered 600 PL/pgSQL‑based ETL/ELT pipelines to streamline complex data processing workflows across multiple PostgreSQL development and production environments.

• A mapping product, carried end to end

  Architected, developed, implemented, supported infrastructure, data processing, and the map application for 2 years non‑stop without any weekends, holidays, or vacations, 10–14 hours a day.

• Quicker answers from a better choice of key

  Adopted UUID v7 time‑ordered identifiers (PostgreSQL 18) as entity keys to reduce B‑tree index fragmentation and speed up queries.

• Modeled 348 tables across 34 Postgres schemas

  Modeled the maritime domain into 348 normalized tables across 34 PostgreSQL schemas — professionals, companies, ships, jobs, reviews and the rest — with SMALLINT lookups and UUID v7 keys.

• Built first‑party analytics inside PostgreSQL

  Built first‑party product analytics in PostgreSQL — 47 functions over partitioned event tables that prune themselves — pseudonymised behind a rotating salt and gated on the visitor's consent.

This work is part of what we offer as Data Pipeline Development (ETL/ELT), GIS & Geospatial Solutions, Data Analytics & BI Dashboards, Database Performance Tuning, Data Governance & Quality, Database Design & Modeling, Backend & API Development and Database Administration (DBA).

Building something that needs this? Get in touch.