Data Engineer (GCP) ID56375
AgileEngine is an Inc. 5000 company that creates award-winning software for Fortune 500 brands and trailblazing startups across 17+ industries. We rank among the leaders in areas like application development and AI/ML, and our people-first culture has earned us multiple Best Place to Work awards.
WHY JOIN US
If you're looking for a place to grow, make an impact, and work with people who care, we'd love to meet you!
ABOUT THE ROLE
WHAT YOU WILL DO
- Build and maintain scalable, distributed, fault-tolerant data pipelines on GCP;
- Develop and manage lakehouse layers and Delta Lake workflows using BigQuery and Dataproc;
- Collaborate with stakeholders across data engineering, compliance, and business teams;
- Design and implement pipelines to acquire, normalise, transform, and release large volumes of financial data;
- Design and implement bitemporal data models on BigQuery for regulatory-grade time-series datasets;
- Build and maintain testing frameworks for data pipelines and transformation logic;
- Own end-to-end solutions including ingestion pipelines, QA workflows, correction management, and audit trails;
- Contribute to shared platform services in a collaborative environment;
- Support implementation of AI solutions including data ingestion, anomaly detection, and semantic search using Vertex AI.
MUST HAVES
- 6–8 years of experience in data engineering;
- Proficiency in Python for data pipelines, transformation logic, and automation;
- Proficiency in SQL with hands-on experience in BigQuery including partitioning, clustering, and time-series queries;
- Experience with Cloud Composer (Apache Airflow) for pipeline orchestration;
- Working knowledge of Dataproc (Apache Spark) for batch ingestion and incremental processing;
- Experience with AI-assisted development tools such as GitHub Copilot or similar;
- Experience with Git version control and collaboration workflows;
- Familiarity with REST APIs for integrations;
- Familiarity with GCP technologies (Cloud Storage, Pub/Sub, Datastream, Cloud Monitoring, IAM, VPC Service Controls);
- Understanding of financial data concepts related to equities and& other asset classes;
- Upper-intermediate English level.
NICE TO HAVES
- Knowledge of data libraries such as pandas or PySpark;
- Experience with columnar storage and time-series analytics tools such as ClickHouse;
- Familiarity with Dataplex for data governance and lineage;
- Understanding of Change Data Capture (CDC) using Datastream;
- Understanding of bitemporal data modeling concepts;
- Knowledge of financial reference data such as equities, fixed income, or corporate actions;
- Experience with BigQuery cost management techniques;
- Experience with CI/CD pipelines and Terraform for infrastructure as code;
- Exposure to LLMs and& Agentic AI using Vertex AI for data-related use cases.
PERKS AND BENEFITS
- Remote work & Local connection: Work where you feel most productive and connect with your team in periodic meet-ups to strengthen your network and connect with other top experts.
- Legal presence in India: We ensure full local compliance with a structured, secure work environment tailored to Indian regulations.
- Competitive Compensation in INR: Fair compensation in INR with dedicated budgets for your personal growth, education, and wellness.
- Innovative Projects: Leverage the latest tech and create cutting-edge solutions for world-recognized clients and the hottest startups.
WHY JOIN US
If you're looking for a place to grow, make an impact, and work with people who care, we'd love to meet you!
ABOUT THE ROLE
WHAT YOU WILL DO
- Build and maintain scalable, distributed, fault-tolerant data pipelines on GCP;
- Develop and manage lakehouse layers and Delta Lake workflows using BigQuery and Dataproc;
- Collaborate with stakeholders across data engineering, compliance, and business teams;
- Design and implement pipelines to acquire, normalise, transform, and release large volumes of financial data;
- Design and implement bitemporal data models on BigQuery for regulatory-grade time-series datasets;
- Build and maintain testing frameworks for data pipelines and transformation logic;
- Own end-to-end solutions including ingestion pipelines, QA workflows, correction management, and audit trails;
- Contribute to shared platform services in a collaborative environment;
- Support implementation of AI solutions including data ingestion, anomaly detection, and semantic search using Vertex AI.
MUST HAVES
- 6–8 years of experience in data engineering;
- Proficiency in Python for data pipelines, transformation logic, and automation;
- Proficiency in SQL with hands-on experience in BigQuery including partitioning, clustering, and time-series queries;
- Experience with Cloud Composer (Apache Airflow) for pipeline orchestration;
- Working knowledge of Dataproc (Apache Spark) for batch ingestion and incremental processing;
- Experience with AI-assisted development tools such as GitHub Copilot or similar;
- Experience with Git version control and collaboration workflows;
- Familiarity with REST APIs for integrations;
- Familiarity with GCP technologies (Cloud Storage, Pub/Sub, Datastream, Cloud Monitoring, IAM, VPC Service Controls);
- Understanding of financial data concepts related to equities and& other asset classes;
- Upper-intermediate English level.
NICE TO HAVES
- Knowledge of data libraries such as pandas or PySpark;
- Experience with columnar storage and time-series analytics tools such as ClickHouse;
- Familiarity with Dataplex for data governance and lineage;
- Understanding of Change Data Capture (CDC) using Datastream;
- Understanding of bitemporal data modeling concepts;
- Knowledge of financial reference data such as equities, fixed income, or corporate actions;
- Experience with BigQuery cost management techniques;
- Experience with CI/CD pipelines and Terraform for infrastructure as code;
- Exposure to LLMs and& Agentic AI using Vertex AI for data-related use cases.
PERKS AND BENEFITS
- Remote work & Local connection: Work where you feel most productive and connect with your team in periodic meet-ups to strengthen your network and connect with other top experts.
- Legal presence in India: We ensure full local compliance with a structured, secure work environment tailored to Indian regulations.
- Competitive Compensation in INR: Fair compensation in INR with dedicated budgets for your personal growth, education, and wellness.
- Innovative Projects: Leverage the latest tech and create cutting-edge solutions for world-recognized clients and the hottest startups.