Big data technologies have been growing exponentially over the past few years and have penetrated into every domain and industry in software development. It has become a core skill for a software engineer. Robust and effective big data pipelines are needed to support the growing volume of data and applications in the big data world. These pipelines have become business critical and help increase revenues and reduce cost.
Do quality big data pipelines happen by magic? High quality designs that are scalable, reliable and cost effective are needed to build and maintain these pipelines.
How do you build an end-to-end big data pipeline that leverages big data technologies and practices effectively to solve business problems? How do you integrate them in a scalable and reliable manner? How do you deploy, secure and operate them? How do you look at the overall forest and not just the individual trees? This course focuses on this skill gap.
What are the topics covered in this course?
We start off by discussing the building blocks of big data pipelines, their functions and challenges.
We introduce a structured design process for building big data pipelines.
We then discuss individual building blocks, focusing on the design patterns available, their advantages, shortcomings, use cases and available technologies.
We recommend several best practices across the course.
We finally implement two use cases for illustration on how to apply the learnings in the course to a real world problem. One is a batch use case and another is a real time use case. Who this course is for:
Big Data Pipeline Designers & Architects Big Data Developers looking to move into Design/Architecture roles Software Architects looking to gain Big Data Experience
Requirements
Big Data Technology Concepts Familiarity with Big Data Technologies like Apache Spark, Apache Kafka and NoSQL Development / Deployment Experience with Big Data Technologies and Pipelines Software Design and Development Experience including Cloud & Microservices
[TutsNode.net] - System Design for Big Data Pipelines/1 - Introduction Expectations/1 - Need for Quality Pipeline Design.mp4
23 MB
[TutsNode.net] - System Design for Big Data Pipelines/1 - Introduction Expectations/2 - Course Coverage and Prerequisites.mp4
21.5 MB
[TutsNode.net] - System Design for Big Data Pipelines/10 - Infrastructure and Deployments/58 - Infrastructure Technologies.mp4
31.5 MB
[TutsNode.net] - System Design for Big Data Pipelines/10 - Infrastructure and Deployments/60 - Processing Jobs Deployments.mp4
33.3 MB
[TutsNode.net] - System Design for Big Data Pipelines/10 - Infrastructure and Deployments/61 - Databases and Queues Deployments.mp4
26.4 MB
[TutsNode.net] - System Design for Big Data Pipelines/10 - Infrastructure and Deployments/62 - Geographical Distribution.mp4
31.7 MB
[TutsNode.net] - System Design for Big Data Pipelines/12 - Serviceability/71 - Pipeline Troubleshooting.mp4
19.6 MB
[TutsNode.net] - System Design for Big Data Pipelines/13 - Use Case I Customer Journey Analytics CJA/72 - Problem Definition for CJA.mp4
29 MB
[TutsNode.net] - System Design for Big Data Pipelines/13 - Use Case I Customer Journey Analytics CJA/74 - Analyze CJA Input Data.mp4
22.1 MB
[TutsNode.net] - System Design for Big Data Pipelines/13 - Use Case I Customer Journey Analytics CJA/76 - Study CJA Pipeline Flowchart.mp4
43 MB
[TutsNode.net] - System Design for Big Data Pipelines/13 - Use Case I Customer Journey Analytics CJA/77 - Create CJA Skeleton Design.mp4
47 MB
[TutsNode.net] - System Design for Big Data Pipelines/13 - Use Case I Customer Journey Analytics CJA/78 - Analyze CJA Scaling.mp4
33.2 MB
[TutsNode.net] - System Design for Big Data Pipelines/13 - Use Case I Customer Journey Analytics CJA/79 - Select Technologies for CJA.mp4
20.9 MB
[TutsNode.net] - System Design for Big Data Pipelines/14 - Use Case II Suspicious Login Alerting SLA/85 - Draw SLA Pipeline Flowchart.mp4
32.6 MB
[TutsNode.net] - System Design for Big Data Pipelines/14 - Use Case II Suspicious Login Alerting SLA/86 - Create SLA Skeleton Design.mp4
26.6 MB
[TutsNode.net] - System Design for Big Data Pipelines/14 - Use Case II Suspicious Login Alerting SLA/87 - Analyze SLA Scaling.mp4
30.4 MB
[TutsNode.net] - System Design for Big Data Pipelines/14 - Use Case II Suspicious Login Alerting SLA/88 - Select SLA Technologies.mp4
20 MB
[TutsNode.net] - System Design for Big Data Pipelines/3 - System Design Process/13 - Analyze Functional Requirements.mp4
24.6 MB
[TutsNode.net] - System Design for Big Data Pipelines/3 - System Design Process/14 - Analyze Pipeline Input.mp4
25.7 MB
[TutsNode.net] - System Design for Big Data Pipelines/3 - System Design Process/17 - Create a Skeleton Design.mp4
20.2 MB
[TutsNode.net] - System Design for Big Data Pipelines/3 - System Design Process/18 - Analyze Scaling.mp4
24.4 MB
[TutsNode.net] - System Design for Big Data Pipelines/3 - System Design Process/19 - Select Technologies.mp4
27.9 MB
[TutsNode.net] - System Design for Big Data Pipelines/4 - Scalable Pipelines Design Principles/22 - Batch vs Realtime Pipelines.mp4
40 MB
[TutsNode.net] - System Design for Big Data Pipelines/4 - Scalable Pipelines Design Principles/23 - Distributed Architectures.mp4
24.5 MB
[TutsNode.net] - System Design for Big Data Pipelines/4 - Scalable Pipelines Design Principles/24 - Microservices based Architectures.mp4
34.2 MB
[TutsNode.net] - System Design for Big Data Pipelines/4 - Scalable Pipelines Design Principles/25 - Batch Pipelines Best Practices.mp4
28.7 MB
[TutsNode.net] - System Design for Big Data Pipelines/4 - Scalable Pipelines Design Principles/26 - Realtime Pipelines Best Practices.mp4
28 MB
[TutsNode.net] - System Design for Big Data Pipelines/4 - Scalable Pipelines Design Principles/27 - Performance Benchmarking for Big Data Pipelines.mp4
29.9 MB
[TutsNode.net] - System Design for Big Data Pipelines/5 - Data Acquisition Design/28 - File Transfer Pattern.mp4
22.4 MB
[TutsNode.net] - System Design for Big Data Pipelines/5 - Data Acquisition Design/29 - Extraction Client Pattern.mp4
19.9 MB
[TutsNode.net] - System Design for Big Data Pipelines/5 - Data Acquisition Design/30 - Ingestion API Pattern.mp4
20.9 MB
[TutsNode.net] - System Design for Big Data Pipelines/5 - Data Acquisition Design/32 - Data Acquisition Design Practices.mp4
21.2 MB
[TutsNode.net] - System Design for Big Data Pipelines/6 - Data Transport Design/33 - Extract Load Pattern.mp4
21.4 MB
[TutsNode.net] - System Design for Big Data Pipelines/6 - Data Transport Design/34 - Request Response Pattern.mp4
23.5 MB
[TutsNode.net] - System Design for Big Data Pipelines/6 - Data Transport Design/35 - Event Streaming Pattern.mp4
28.1 MB
[TutsNode.net] - System Design for Big Data Pipelines/7 - Data Processing Transformation Design/37 - Data Processing Patterns.mp4
24.7 MB
[TutsNode.net] - System Design for Big Data Pipelines/7 - Data Processing Transformation Design/38 - Distributed Processing with Big Data.mp4
19.6 MB
[TutsNode.net] - System Design for Big Data Pipelines/7 - Data Processing Transformation Design/39 - Batch Processing Design Practices Part 1.mp4
37 MB
[TutsNode.net] - System Design for Big Data Pipelines/7 - Data Processing Transformation Design/40 - Batch Processing Design Practices Part 2.mp4
27.5 MB
[TutsNode.net] - System Design for Big Data Pipelines/7 - Data Processing Transformation Design/41 - Stream Processing Design Practices.mp4
29.7 MB
[TutsNode.net] - System Design for Big Data Pipelines/7 - Data Processing Transformation Design/43 - Input and Output Considerations for Processing.mp4
26.6 MB
[TutsNode.net] - System Design for Big Data Pipelines/7 - Data Processing Transformation Design/44 - Processing Engine Technologies.mp4
24.8 MB
[TutsNode.net] - System Design for Big Data Pipelines/8 - Storage Design/45 - Distributed File System Pattern.mp4
29 MB
[TutsNode.net] - System Design for Big Data Pipelines/8 - Storage Design/46 - Relational Database Pattern.mp4
21.6 MB
[TutsNode.net] - System Design for Big Data Pipelines/8 - Storage Design/47 - Document Database Pattern.mp4
19.1 MB
[TutsNode.net] - System Design for Big Data Pipelines/8 - Storage Design/51 - Data Storage Design Practices 1.mp4
21.1 MB
[TutsNode.net] - System Design for Big Data Pipelines/9 - Serving Design/53 - Query Interface Pattern.mp4
26.3 MB
[TutsNode.net] - System Design for Big Data Pipelines/9 - Serving Design/54 - Serving API Pattern.mp4
20.2 MB
[TutsNode.net] - System Design for Big Data Pipelines/9 - Serving Design/57 - Data Serving Design Practices.mp4
19.4 MB
Alternative Torrents for 'System Design for Big Data Pipelines'.
