Senior Data Engineer

На данный момент Senior Software Engineer.

Мои компетенции и опыт

Ethan Lim

Singapore | нужен доступ к резюме | нужен доступ к резюме | нужен доступ к резюме

Summary

Experienced Senior Data Engineer with 7+ years in designing and implementing scalable data storage and processing systems using a diverse tech stack including Python, Apache Spark, PostgreSQL, and ClickHouse. Proven ability to develop integration flows, manage OLAP/OLTP data systems, and automate complex data processes in fast-paced environments. Skilled in optimizing data workflows and maintaining high system reliability while collaborating across teams to drive impactful business decisions. Ready to deliver immediate value with expert-level proficiency in large-scale data processing and cloud infrastructure.

Experience

Rescode | Senior Software Engineer May 2022 ~ Present

• Spearheaded the migration of large data pipelines to Apache Spark, optimizing processing speed by 50%.
• Integrated Apache Kafka and Apache Flink to improve data flow, achieving real-time data processing for user interactions.
• Built and optimized OLAP/OLTP systems using PostgreSQL, increasing query speed by 30%.
• Developed complex batch processing workflows using Apache NiFi, reducing data latency by 40%.
• Enhanced data pipelines with Apache Airflow for orchestration, reducing task failure rates by 25%.
• Led the design and implementation of scalable data storage solutions across AWS cloud infrastructure.
• Managed and maintained production databases in ClickHouse and Cassandra, ensuring нужен доступ к резюме % uptime.
• Worked closely with data analysts and business stakeholders to align data solutions with business objectives.
• Optimized data processing jobs, leading to a 20% improvement in overall system performance.
• Documented technical processes and workflows for internal teams using Confluence, improving knowledge sharing.
• Assisted the technical support team in troubleshooting and resolving data-related incidents.
• Collaborated with DevOps to enhance CI/CD pipelines, leading to a 30% reduction in deployment time.
• Introduced automated data integrity checks, ensuring consistency and reliability of production data.
• Mentored junior engineers, providing guidance on best practices for data processing and storage.
• Contributed to cross-functional teams, driving continuous improvements in data systems and business outcomes.

Doodle Labs | Senior Full Stack Engineer Apr 2019 ~ Apr 2022

• Built and optimized real-time data systems using Apache Kafka and Apache Spark for e-commerce applications.
• Implemented data storage solutions using PostgreSQL and Redis, reducing response time by 25%.
• Developed APIs for seamless data integration, enabling faster data exchange across platforms.
• Introduced automated ETL pipelines, reducing manual data processing efforts by 40%.
• Integrated batch processing systems with Apache NiFi, optimizing data workflow across teams.
• Maintained and scaled cloud infrastructure using AWS, ensuring high availability for data services.
• Enhanced data security protocols, ensuring compliance with GDPR and improving overall system security.
• Led the design and development of internal tools to support business operations and data analysis.
• Collaborated with the product team to define data requirements and ensure efficient implementation.
• Optimized PostgreSQL queries, improving data retrieval performance by 35%.
• Managed deployments of data processing systems using Docker and Kubernetes, ensuring system scalability.
• Provided support for troubleshooting production data issues, improving resolution time by 30%.
• Led internal training sessions for junior engineers on data engineering best practices.

Wise | Full Stack Engineer May 2015 ~ Feb 2019

• Built and optimized databases for a financial platform, improving data access speed by 20%.
• Created and maintained data pipelines for transaction processing and reporting.
• Integrated Redis and PostgreSQL to ensure quick data retrieval and cache efficiency.
• Participated in optimizing batch processing jobs using Apache Spark, reducing job completion time by 15%.
• Developed and deployed APIs to support data interaction with various internal and external systems.
• Assisted in creating a scalable data warehouse solution, improving reporting speed by 25%.
• Utilized Python for data manipulation, leading to a 20% increase in data processing speed.
• Developed automation scripts for data backups, ensuring data integrity and availability.
• Collaborated closely with cross-functional teams to deliver data-driven features.
• Implemented continuous integration practices to ensure smooth and rapid development cycles.

Education

National University of Singapore Aug 2009 ~ Jul 2013

Bachelor’s Degree in Computer Science

Skills

Programming Languages & Frameworks: Python, Apache Spark, Apache Flink, Apache Airflow, Apache NiFi
Data Engineering: Data Integration, Data Modeling, OLAP, OLTP, Batch Processing, CDC
Databases: PostgreSQL, Cassandra, ClickHouse, MS SQL, Redis
Cloud Platforms & Infrastructure: AWS, GCE, Docker, Kubernetes
Data Processing Tools: Apache Kafka, Apache Iceberg, Apache Paimon
Automation & CI-CD: Airflow, Jenkins, GitHub Actions
Collaboration / Agile Tools: Jira, Confluence, Slack, Git, GitFlow