Big Data Lead
We are seeking a highly experienced Big Data Lead to design, develop, and manage scalable data processing solutions using the Hadoop ecosystem and modern data engineering technologies. The ideal candidate will have strong hands-on expertise in Apache Spark, Scala, Python, Hadoop, and cloud platforms (AWS or GCP). This role requires technical leadership in building robust data pipelines, supporting large-scale analytics, and mentoring data engineering teams.
Key Responsibilities
Lead the design and development of scalable big data platforms and data pipelines.
Develop and optimize batch and real-time data processing solutions using Apache Spark.
Work with Spark RDD APIs and implement distributed data processing applications.
Build and maintain ETL workflows for processing large volumes of structured and unstructured data.
Utilize Hadoop ecosystem tools such as Hadoop, Hive, Pig, and Oozie.
Implement streaming data pipelines using technologies like Kafka.
Collaborate with data scientists, analysts, and engineering teams to support analytics and business intelligence requirements.
Ensure performance tuning, scalability, and reliability of big data systems.
Provide technical guidance and mentorship to junior data engineers.
Work with cloud platforms (AWS or GCP) to deploy and manage data infrastructure.
Maintain and support Linux-based environments for data processing systems.
Required Skills
10+ years of experience in Big Data, Data Engineering, or related roles.
Strong hands-on experience with Hadoop ecosystem technologies including Hadoop, Hive, Pig, and Oozie.
Extensive experience with Apache Spark and strong knowledge of Spark RDD APIs.
Proficiency in Scala and Python programming.
Experience with Core Java.
Strong knowledge of SQL and scripting languages.
Hands-on experience with Kafka or similar streaming frameworks.
Strong experience working in Linux environments.
Data & Analytics Knowledge
Strong understanding of Data Warehousing concepts.
Experience with Data Modeling techniques.
Familiarity with data visualization and analytics tools such as Tableau or R.
Cloud Experience
Hands-on experience with AWS or GCP cloud platforms.
Experience building data engineering workflows and big data solutions in cloud environments.
Preferred Qualifications
Experience designing real-time streaming architectures.
Strong experience in performance tuning and optimization for Spark and Hadoop.
Prior experience leading data engineering teams or projects.
Apply tot his job
Apply To this Job
Key Responsibilities
Lead the design and development of scalable big data platforms and data pipelines.
Develop and optimize batch and real-time data processing solutions using Apache Spark.
Work with Spark RDD APIs and implement distributed data processing applications.
Build and maintain ETL workflows for processing large volumes of structured and unstructured data.
Utilize Hadoop ecosystem tools such as Hadoop, Hive, Pig, and Oozie.
Implement streaming data pipelines using technologies like Kafka.
Collaborate with data scientists, analysts, and engineering teams to support analytics and business intelligence requirements.
Ensure performance tuning, scalability, and reliability of big data systems.
Provide technical guidance and mentorship to junior data engineers.
Work with cloud platforms (AWS or GCP) to deploy and manage data infrastructure.
Maintain and support Linux-based environments for data processing systems.
Required Skills
10+ years of experience in Big Data, Data Engineering, or related roles.
Strong hands-on experience with Hadoop ecosystem technologies including Hadoop, Hive, Pig, and Oozie.
Extensive experience with Apache Spark and strong knowledge of Spark RDD APIs.
Proficiency in Scala and Python programming.
Experience with Core Java.
Strong knowledge of SQL and scripting languages.
Hands-on experience with Kafka or similar streaming frameworks.
Strong experience working in Linux environments.
Data & Analytics Knowledge
Strong understanding of Data Warehousing concepts.
Experience with Data Modeling techniques.
Familiarity with data visualization and analytics tools such as Tableau or R.
Cloud Experience
Hands-on experience with AWS or GCP cloud platforms.
Experience building data engineering workflows and big data solutions in cloud environments.
Preferred Qualifications
Experience designing real-time streaming architectures.
Strong experience in performance tuning and optimization for Spark and Hadoop.
Prior experience leading data engineering teams or projects.
Apply tot his job
Apply To this Job