Data Engineer

Mô tả Công việc

1. Data pipeline development and optimization

· Design, build, and maintain efficient, scalable ETL/ELT pipelines for structured and unstructured data.

· Implement modular, reusable Python-based processing scripts and manage job orchestration using Power Automate, Linux cron, or other scheduling frameworks.

· Continuously monitor and optimize pipeline performance for speed and resource utilization.

2. Cloud infrastructure and cost management

· Deploy and manage data infrastructure leveraging AWS services (e.g. Lambda, Glue, S3, RDS, Athena).

· Ensure system uptime, data accessibility, and high availability across environments.

· Proactively optimize storage and compute costs through monitoring, automation, and right-sizing of resources.

3. Automation, CI/CD & DevOps practices

· Automate deployment workflows using GitLab CI/CD pipelines.

· Maintain a clean, testable codebase with proper version control and documentation.

· Introduce best practices for environment provisioning, rollback, and automated testing.

4. Data quality, security & governance

· Implement robust data validation, logging, and monitoring practices across pipelines.

· Ensure data privacy, integrity, and lifecycle compliance in collaboration with IT security policy.

· Maintain documentation including data dictionaries, flow diagrams, and operational runbooks.

5. Cross team technical support & enablement

· Collaborate with IT, software, and business system teams to integrate backend data solutions.

· Ensure that platform components (pipelines, storage, compute) support the broader application and reporting ecosystem.

· Provide technical guidance on data ingestion, transformation, and storage patterns.

Yêu Cầu Công Việc

Required Competencies

· Strong programming skills in Python, with experience building modular and reusable components for data workflows

· Proficient in SQL for data manipulation and performance tuning across relational databases (PostgreSQL, MySQL)

· Hands-on experience with AWS services such as Lambda, Glue, S3, RDS, and Athena, with a focus on cost and performance optimization

· Fluent in CI/CD and automation frameworks, particularly using GitLab pipelines, Power Automate, and scripting on Linux/Windows environments

· Solid understanding of data pipeline orchestration (e.g., cron, Airflow, Power Automate) and containerization (e.g., Docker)

· Knowledge of data governance, privacy, and quality control, including validation, monitoring, and alerting mechanisms

· Familiar with version control, infrastructure documentation, and maintaining data dictionaries or runbooks

· Demonstrated ability to collaborate with cross-functional technical teams, supporting data infrastructure integration and maintenance

Qualification and Experience Required

· Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related technical field

· Professional certification in data engineering, cloud architecture, or AWS (e.g., AWS Certified Data Analytics or Solutions Architect) is an advantage

· Minimum 3 years of hands-on experience in a data engineering or backend data infrastructure role

· Proven track record in building and maintaining production-grade ETL/ELT pipelines using Python and SQL

· Demonstrated experience managing cloud-based infrastructure, with a strong understanding of cost optimization, monitoring, and scalability (preferably on AWS)

· Experience working with DevOps practices, CI/CD pipelines (e.g., GitLab), and automation tools for deployment and workflow orchestration

· Familiarity with data quality, security, and governance best practices

· Experience in a multi-business or regional group environment is a plus