1. Data Management & Development
- Design, build, and operate data platforms such as Hadoop, Yarn, Spark (or alternatives like Kubernetes, Delta Lake (Iceberg), S3 on-prem).
- Utilize ETL and orchestration tools including ODI, Airflow, or equivalent solutions.
- Apply Git, Jenkins, and CI/CD practices in data engineering workflows.
- Develop and implement Data Governance frameworks (e.g., DataHub, OpenMetadata).
- Demonstrate strong expertise in PL/SQL (Oracle preferred).
- Design, manage, and optimize Data Warehouse, Data Lake, and Data Mart systems to support analytics and reporting needs.
2. Data Quality & Integration
- Standardize and ensure data quality through cleansing, reconciliation, periodic checks, and validation logic.
- Collaborate with IT teams and business units to integrate data from multiple sources and systems.
- Build and optimize ETL/ELT pipelines to extract, transform, and load data from databases, APIs, and files.
3. Data Platform Operations & Optimization
- Ensure data integrity, availability, security, and performance across all data systems.
- Optimize queries and large-scale data processing (big data) to improve performance and reduce processing time.
- Develop automated data validation processes (e.g., schema checks, anomaly detection).
4. Team Management & Capability Development
- Lead, coach, and develop a team of Data Engineers.
- Establish internal processes, promote knowledge sharing, and standardize documentation and analytics practices.