Senior Data Engineer (Remote, AWS, Databricks, EMR)

About the Company

Are you ready to take your career to the next level? At Blue Ridge, we're a fantastic SaaS company specializing in supply chain planning. We focus on demand planning, replenishment, and integrated business planning solutions. With over 220 clients around the globe—primarily in the US and Norway—we're growing rapidly at over 20% annually!

Now, we’re excited to open a new R&D office in the EU and are looking for a talented Senior Data Engineer to join our growing team. If you’re eager to work with cutting-edge technology and tackle complex challenges, this could be the perfect opportunity for you!

About the Role

As a Data Engineer, you’ll be a key player in driving our company’s technological advancements and building a scalable data platform. Our current setup includes .NET, C#, and traditional databases, but we’re thrilled to be prototyping with Amazon EMR and Databricks to tackle our growing data needs and boost scalability by an impressive 16 to 20 times! This role offers you the unique opportunity to work on both legacy systems and cutting-edge cloud platforms, helping shape the future of our supply chain solutions.

You’ll be collaborating closely with a fresh team of 2 Data Engineers based in India (GMT+5:30), and we’re also looking to hire 2-3 Senior Data Engineers in the EU over the next year.

In this position, you’ll report to our VP of Engineering, who operates in the EST/EDT time zone.

Position Details:

• Role:Senior Data engineer
• Location: Remote (We’re looking for candidates in Romania, Czech Republic, Poland, Georgia, or Serbia only)
• Employment: Service Agreement (B2B contract; you’ll need a legal entity to sign)
• Start Date: ASAP (October - November, 2024)
• Salary: $5,500 - $8,000 USD per month GROSS (fixed income, paid via SWIFT)
• Working Hours: 11 AM to 7 PM local time
• Time Overlaps: Expect meetings with teams in GMT+5:30, GMT-5, plus occasional meetings with the VP of Engineering in EST/EDT and CTO in GMT-8.

What You’ll Be Doing

• Transitioning Legacy Systems: You’ll help us move our old systems into a modern tech stack that includes .NET, C#, and various databases.
• Designing a Scalable Data Platform: You’ll be in charge of designing and building a highly scalable, fault-tolerant data platform optimized for distributed computing and large-scale data processing using Amazon EMR, Databricks, and other big data technologies.
• Implementing Data Pipelines: You’ll implement data pipelines and ETL/ELT processes using distributed computing frameworks to efficiently ingest, transform, and load massive datasets from various sources.
• Leveraging Cloud Data Platforms: You’ll leverage cloud data platforms to enable seamless data sharing, near-zero maintenance, and fast analytics on both structured and semi-structured data.
• Optimizing Performance: Your role will involve optimizing distributed computing jobs and queries to ensure maximum performance and cost efficiency.
• Partnering with Data Experts: You’ll collaborate with data scientists, machine learning engineers, and software developers to understand data requirements and create innovative solutions that power our GenAI applications.
• Collaborating with Diverse Teams: You’ll work closely with our diverse teams to design, implement, and maintain sustainable, high-performance data processing and integration systems.
• Implementing Best Practices: You’ll help us implement data governance, security, and compliance best practices to keep our data safe and sound.
• Driving Innovation: You’ll partner with global teams to drive innovation in supply chain technology, ensuring we stay ahead of the curve.
• Mentoring Junior Engineers: Finally, you’ll provide guidance on distributed computing architecture and mentor junior data engineers, sharing your knowledge and experience to help them grow.

What We’re Looking For

We’re looking for someone who has:

• Education: A Bachelor’s or Master’s degree in Computer Science or a related field.
• Experience: Over 5 years as a Data Engineer with a strong background in big data technologies.
• Cloud Experience: At least 7 years in cloud architecture, with extensive expertise in AWS (this is a must!). Experience with GCP or Azure is a bonus.
• A solid understanding of AWS architecture, services, networking, and security—think EMR, IAM, VPC, EKS, ALB, and Lambda.
• Strong proficiency in SQL, Python, and data modeling techniques.
• Deep expertise in distributed computing principles and frameworks (e.g., Apache Spark, Apache Airflow), streaming (Kafka), and optimizing jobs for scale and efficiency.
• Extensive experience executing complex functional business application logic.
• Hands-on experience with developing and deploying distributed computing applications using cloud-based platforms (AWS EMR).
• A strong understanding of cloud data platform architectures and best practices for ELT/ETL, data sharing, and query optimization (e.g., AWS Athena, AWS Glue, Azure Synapse Analytics, or equivalents).
• Experience enabling application engineers to build applications leveraging the data platform through APIs and abstractions.
• Knowledge of data modeling, data warehousing, and schema design.
• A solid grasp of .NET code, project structure, and how typical applications are packaged, developed, and deployed.
• Excellent problem-solving and communication skills.
• You have advanced English skills.

Highly Desired Certifications

• Cloud Certifications: AWS, GCP, or Azure certifications (like AWS Certified Solutions Architect - Professional or GCP Professional Cloud Architect) are significant pluses. Specifically, AWS certifications that would help you succeed in this role include:
  • AWS Certified Solutions Architect - Associate or Professional
  • AWS Certified Developer - Associate
  • AWS Certified Big Data - Specialty
• Additional Certifications: Certifications in Databricks, Apache Spark, Airflow, and data modeling-related fields are highly valued.

Other Requirements:

• Remote Work: You can work remotely while maintaining high productivity.
• Time Zone Overlaps: You’re comfortable with meetings that overlap with teams in GMT+5:30, GMT-5, plus occasional meetings with the VP of Engineering in EST/EDT and CTO in GMT-8.
• Legal Entity: You need to have a legal entity to sign the B2B contract.

Why Join Us?

• Innovative Projects: Work with cutting-edge cloud and big data technologies like Databricks and Amazon EMR to solve complex scaling challenges.
• Global Team: Collaborate with a diverse team across the US, India, and Europe on meaningful projects that drive real-world solutions.

Recruitment Process

1. Initial Interview: up to 60 minutes with HR, including a self-assessment form.
2. Test Assignment: up to 90 minutes on iMocha platform (Data Structures - Graph data structure, Array and String manipulation - All in Python, with a few MCQ questions on Spark).
3. Technical Interviews: Technical Interviews (2 rounds):
   • Platform/Application Architect: 45-60 minutes
   • Lead Engineer: 45-60 minutes
4. Managerial Interview: VP of Engineering: 45-60 minutes (Discuss work schedules, platform functionality, company goals, and vision).
5. Final Interview: CTO: Up to 45 minutes (Focus on your technical experience, accomplishments, and career goals)
6. Optional final sync with CTO (pre-offer discussion).

(All calls and meetings will be conducted via MS Teams)

Got Questions or Interested? Let’s Connect!

If this role feels like the perfect fit for you or if you have any questions, let’s schedule an exploratory call this week. We’d love to chat about how your skills align with this exciting opportunity!