As a Senior Data Engineer, you will design and build scalable data pipelines and ensure reliable data infrastructure while collaborating with data scientists. Your role includes transforming datasets, implementing ETL pipelines, and employing cloud-based solutions to support data architecture projects.
Our client is currently undergoing a transition from the Lucernex database hosted on Google Cloud Platform (GCP) to an Azure-based environment using Databricks. We are seeking a skilled Data Migration Specialist to support this critical migration effort. The role will focus on data extraction, transformation, and loading (ETL) across different data platforms, ensuring smooth and reliable migration processes while maintaining data quality. If you have expertise in data ingestion, integration, migration, and cloud-based environments, we'd love to hear from you..
Requirements
- Strong experience in data migration, specifically in moving data across platforms (GCP to Azure).
- Expertise in handling ETL processes, particularly using tools like Altova MapForce.
- Experience working with databases such as SQL Server.
- Familiarity with cloud-based platforms, especially GCP and Azure.
- Knowledge of DevOps tools like Bitbucket and Jenkins for CI/CD processes.
- Ability to work with various data formats (CSV, screenshots, etc.) and create data mappings between systems.
- Strong problem-solving skills with an emphasis on data quality and integrity.
- Experience in environment management and administering cloud platforms.
- Excellent communication skills to collaborate with clients on data mapping and migration requirements.
Responsibilities
- Data Ingestion: Extract data from various source platforms and systems.
- Integration: Combine and transform data from multiple sources to ensure consistency and usability.
- Data Quality: Ensure the accuracy, completeness, and reliability of the data throughout the migration process.
- Migration: Facilitate the movement of data to the destination storage, ensuring proper handling and processing.
- Testing: Validate that the processed data meets business requirements and is fit for purpose.
- Environment Management: Administer the platforms and tools, ensuring the smooth operation of the migration process and maintaining efficient workflows.
Similar Jobs
Big Data • Fitness • Healthtech • Software • Analytics • Energy
The Data Engineer at Arcadia will design, build and test data connectors and ingestion pipelines, contributing to the integration of client claim and clinical data platforms. Responsibilities include peer code reviews, sprint planning, and maintaining JIRA tasks. The role requires problem solving for coding and data analysis issues within a dynamic healthcare environment.
Fintech • Machine Learning • Payments • Software • Financial Services
As a Lead Data Engineer at Capital One, you will design, develop, and support technical solutions across Agile teams, focusing on data management, cloud-based applications, and mentoring. You'll utilize modern programming languages and technologies while collaborating with product managers to deliver impactful solutions in the financial sector.
Top Skills:
AWSCassandraEmrGCPGurobiHadoopHiveJavaKafkaLinuxMapreduceAzureMongodbMySQLNoSQLOpen Source RdbmsPythonRedshiftScalaSnowflakeSparkUnix
Consumer Web • Digital Media • Information Technology • News + Entertainment • Social Media
The Lead Data Platform Engineer will scale a high-volume data platform, managing ETL processes, data warehousing, and governance while mentoring the data platform team and collaborating with analytics and machine learning teams for infrastructure support.
What you need to know about the Seattle Tech Scene
Home to tech titans like Microsoft and Amazon, Seattle punches far above its weight in innovation. But its surrounding mountains, sprinkled with world-famous hiking trails and climbing routes, make the city a destination for outdoorsy types as well. Established as a logging town before shifting to shipbuilding and logistics, the Emerald City is now known for its contributions to aerospace, software, biotech and cloud computing. And its status as a thriving tech ecosystem is attracting out-of-town companies looking to establish new tech and engineering hubs.
Key Facts About Seattle Tech
- Number of Tech Workers: 287,000; 13% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Amazon, Microsoft, Meta, Google
- Key Industries: Artificial intelligence, cloud computing, software, biotechnology, game development
- Funding Landscape: $3.1 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Madrona, Fuse, Tola, Maveron
- Research Centers and Universities: University of Washington, Seattle University, Seattle Pacific University, Allen Institute for Brain Science, Bill & Melinda Gates Foundation, Seattle Children’s Research Institute
