Automated data processing in Python and PL/pgSQL
Automated data processing tasks using Shell scripting, PL/pgSQL,
Python, and Transact‑SQL, increasing productivity and efficiency.
2015 – 2022

Situation. There was a steady load of recurring data‑processing work
being done by hand. Manual data work has two problems at once: it
eats time, and it's inconsistent — do the same task by hand enough
times and it'll get done slightly differently, and some of those
differences are errors.

Task. The aim was to automate these tasks, both to get the time back
and to make them reliable.

Action. The data‑processing work got automated across the databases
and systems it touched, using whatever fit the job — Shell scripting
for the glue, PL/pgSQL and Transact‑SQL down in the databases,
Python where it needed more than SQL could give. Manual steps got
replaced with jobs that ran the same way every time, which is the
whole point: a script doesn't get bored, doesn't skip a step, and
doesn't do it differently on a Friday afternoon.

Result. Productivity and efficiency both went up, the manual effort
came off people's plates, and the data processing became consistent
and dependable instead of a source of small, recurring errors.

Categories:
1Databases	/categories/databases/	engineer.company	70
1PostgreSQL	/categories/postgresql/	engineer.company	70
1SQL	/categories/sql/	engineer.company	70
1Data Engineering	/categories/data-engineering/	engineer.company	70
1Data Pipelines (ETL/ELT)	/categories/data-pipelines/	engineer.company	70
1Automation & CI/CD	/categories/automation/	engineer.company	70
1Python	/categories/python/	engineer.company	70

Services:
1Database Administration (DBA)	/services/database-administration/	engineer.company	70
1Data Pipeline Development (ETL/ELT)	/services/data-pipeline-development/	engineer.company	70
1DevOps & CI/CD Automation	/services/devops-cicd/	engineer.company	70

hThis page on the web	URL:https://engineer.company/portfolio/automated-data-processing-in-python-and-pl-pgsql/	engineer.company	70
