This course is designed to equip data engineers with the skills to build scalable and efficient data pipelines using Scala and Spark. Data engineers will learn best practices for development, testing, and deployment in cloud environments, with a focus on optimizing performance and ensuring data quality. The course provides the necessary tools to transform raw data into actionable insights, making it highly relevant in today’s data-driven world.
Data Engineering with Scala and Spark
您将学到什么
Set up a development environment for building data pipelines in Scala
Use Spark DataFrames, Datasets, and SQL with Scala for data processing
Profile and clean data using Deequ for improved data quality
您将获得的技能
要了解的详细信息
可分享的证书
添加到您的领英档案
最近已更新！
March 2026
作业
13 项作业
授课语言：英语（English）
了解顶级公司的员工如何掌握热门技能
该课程共有13个模块
In this section, we explore functional programming, higher-order functions, polymorphic functions, and pattern matching in Scala for data engineering applications.
涵盖的内容
2个视频6篇阅读材料1个作业
In this section, we explore cloud-based and local environments for data engineering pipelines, focusing on setup processes, trade-offs, and practical applications.
涵盖的内容
1个视频5篇阅读材料1个作业
In this section, we explore Apache Spark's APIs, focusing on DataFrame and Dataset for distributed data processing.
涵盖的内容
1个视频3篇阅读材料1个作业
In this section, we explore using Spark JDBC API for database access, designing database interfaces, and performing operations with configuration loading.
涵盖的内容
1个视频3篇阅读材料1个作业
In this section, we explore object stores, data lakes, and lakehouses, focusing on their roles in managing large-scale data workflows efficiently.
涵盖的内容
1个视频6篇阅读材料1个作业
In this section, we explore Spark transformations, aggregations, joins, and window functions to enhance data processing for BI and analytics. Key concepts include efficient data manipulation and pipeline development.
涵盖的内容
1个视频4篇阅读材料1个作业
In this section, we explore Deequ for implementing data quality checks, analyzing completeness and accuracy, and defining constraints to ensure reliable data pipelines.
涵盖的内容
1个视频3篇阅读材料1个作业
In this section, we explore test-driven development, static code analysis, and linting to improve code quality, maintainability, and consistency in data engineering projects.
涵盖的内容
1个视频4篇阅读材料1个作业
In this section, we explore CI/CD practices with GitHub to automate Scala data pipeline workflows, focusing on GitHub Actions, version control, and reliable deployment processes.
涵盖的内容
1个视频4篇阅读材料1个作业
In this section, we explore data pipeline orchestration using tools like Airflow, Argo, Databricks, and Azure Data Factory. We focus on workflow design, task management, and real-world implementation strategies.
涵盖的内容
1个视频6篇阅读材料1个作业
In this section, we analyze Spark UI metrics to identify performance issues, optimize data shuffling, and right-size compute resources for efficient data processing.
涵盖的内容
1个视频4篇阅读材料1个作业
In this section, we explore building batch pipelines using Spark and Scala, focusing on medallion architecture, data ingestion, transformation, and orchestration for scalable data processing.
涵盖的内容
1个视频5篇阅读材料1个作业
In this section, we explore building real-time data pipelines using Spark, Scala, and Kafka for IoT applications. Key concepts include data ingestion, transformation, and serving layer design.
涵盖的内容
1个视频4篇阅读材料1个作业
位教师
提供方
从 Data Management 浏览更多内容
状态：免费试用Duke University
状态：免费试用
人们为什么选择 Coursera 来帮助自己实现职业发展
Felipe M.
自 2018开始学习的学生
''能够按照自己的速度和节奏学习课程是一次很棒的经历。只要符合自己的时间表和心情，我就可以学习。'
Jennifer J.
自 2020开始学习的学生
''我直接将从课程中学到的概念和技能应用到一个令人兴奋的新工作项目中。'
Larry W.
自 2021开始学习的学生
''如果我的大学不提供我需要的主题课程，Coursera 便是最好的去处之一。'
Chaitanya A.
''学习不仅仅是在工作中做的更好：它远不止于此。Coursera 让我无限制地学习。'
常见问题
Yes, you can preview the first video and view the syllabus before you enroll. You must purchase the course to access content not included in the preview.
If you decide to enroll in the course before the session start date, you will have access to all of the lecture videos and readings for the course. You’ll be able to submit assignments once the session starts.
Once you enroll and your session begins, you will have access to all videos and other resources, including reading items and the course discussion forum. You’ll be able to view and submit practice assessments, and complete required graded assignments to earn a grade and a Course Certificate.
更多问题
提供助学金，