Skip to content

Commit ed2a251

Browse files
nilbuildJavi Canales
andauthored
Add content to Data Engineer Roadmap (#9016)
* Add basic content * add content to data engineer roadmap * add content to DE roadmap and fix some typos in content appearing in several roadmaps * batch of new content for data engineer roadmap * new batch of content from DE roadmap * new batch in DE roadmap with 25 contents * add 30 new content for DE roadmap * new 30 contents for DE roadmap * add last batch of content for DE roadmap. Ready to PR * add 4 missing contents * clean typo in de roadmap --------- Co-authored-by: Javi Canales <javicanales@Dans-Laptop.local>
1 parent 449e8f1 commit ed2a251

186 files changed

Lines changed: 1729 additions & 186 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,8 @@
1-
# A/B Testing
1+
# A/B Testing
2+
3+
A/B testing is a way to compare two versions of something to see which one works better. You split your audience into two groups, one sees version A, the other sees version B — and then you measure which version gets better results, like more clicks, sales, or sign-ups. This helps you make decisions based on real data instead of guesses.
4+
5+
Visit the following resources to learn more:
6+
7+
- [@article@A software engineer's guide to A/B testing](https://posthog.com/product-engineers/ab-testing-guide-for-engineers)
8+
- [@video@A/B Testing for Beginners](https://www.youtube.com/watch?v=VpTlNRUcIDo)
Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,8 @@
1-
# Amazon EC2 ( Compute)
1+
# Amazon EC2 ( Compute)
2+
3+
Amazon Elastic Compute Cloud (EC2) is a web service that provides secure, resizable compute capacity in the cloud. It is designed to make web-scale cloud computing easier for developers. EC2’s simple web service interface allows you to obtain and configure capacity with minimal friction. EC2 enables you to scale your compute capacity, develop and deploy applications faster, and run applications on AWS's reliable computing environment. You have the control of your computing resources and can access various configurations of CPU, Memory, Storage, and Networking capacity for your instances.
4+
5+
Visit the following resources to learn more:
6+
7+
- [@official@EC2 - User Guide](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html)
8+
- [@video@Introduction to Amazon EC2](https://www.youtube.com/watch?v=eaicwmnSdCs)
Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,7 @@
1-
# Amazon RDS (Database)
1+
# Amazon RDS (Database)
2+
3+
Amazon RDS (Relational Database Service) is a web service from Amazon Web Services. It's designed to simplify the setup, operation, and scaling of relational databases in the cloud. This service provides cost-efficient, resizable capacity for an industry-standard relational database and manages common database administration tasks. RDS supports six database engines: Amazon Aurora, PostgreSQL, MySQL, MariaDB, Oracle Database, and SQL Server. These engines give you the ability to run instances ranging from 5GB to 6TB of memory, accommodating your specific use case. It also ensures the database is up-to-date with the latest patches, automatically backs up your data and offers encryption at rest and in transit.
4+
5+
Visit the following resources to learn more:
6+
7+
- [@official@Amazon RDS](https://aws.amazon.com/rds/)
Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,7 @@
1-
# Amazon RDS (Database)
1+
# Amazon RDS (Database)
2+
3+
Amazon RDS (Relational Database Service) is a web service from Amazon Web Services. It's designed to simplify the setup, operation, and scaling of relational databases in the cloud. This service provides cost-efficient, resizable capacity for an industry-standard relational database and manages common database administration tasks. RDS supports six database engines: Amazon Aurora, PostgreSQL, MySQL, MariaDB, Oracle Database, and SQL Server. These engines give you the ability to run instances ranging from 5GB to 6TB of memory, accommodating your specific use case. It also ensures the database is up-to-date with the latest patches, automatically backs up your data and offers encryption at rest and in transit.
4+
5+
Visit the following resources to learn more:
6+
7+
- [@official@Amazon RDS](https://aws.amazon.com/rds/)
Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,8 @@
1-
# Amazon Redshift
1+
# Amazon Redshift
2+
3+
Amazon Redshift is a cloud-based data warehouse service from Amazon that lets you store and analyze large amounts of data quickly. It’s designed for running complex queries on huge datasets, so businesses can use it to turn raw data into useful reports and insights. You can load data into Redshift from many sources, and then use SQL to explore it, just like you would with a regular database — but it’s optimized to handle much bigger data and run faster.
4+
5+
Visit the following resources to learn more:
6+
7+
- [@official@Amazon Redshift](https://aws.amazon.com/redshift/)
8+
- [@video@Getting Started with Amazon Redshift - AWS Online Tech Talks](https://www.youtube.com/watch?v=dfo4J5ZhlKI)
Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,7 @@
1-
# Apache Airflow
1+
# Apache Airflow
2+
3+
Apache Airflow is an open-source tool that helps you schedule, organize, and monitor workflows. Think of it like a to-do list for your data tasks, but smarter — you can set tasks to run in a specific order, track their progress, and see what happens if something fails. It’s often used for automating data pipelines so that data moves, gets processed, and is ready for use without manual work.
4+
5+
Visit the following resources to learn more:
6+
7+
- [@official@Apache Airflow](https://airflow.apache.org/)
Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,7 @@
1-
# Apache Hadoop YARN
1+
# Apache Hadoop YARN
2+
3+
Apache Hadoop YARN (Yet Another Resource Negotiator) is the part of Hadoop that manages resources and runs jobs on a cluster. It has a ResourceManager that controls all cluster resources and an ApplicationMaster for each job that schedules and runs tasks. YARN lets different tools like MapReduce and Spark share the same cluster, making it more efficient, flexible, and reliable.
4+
5+
Visit the following resources to learn more:
6+
7+
- [@video@Hadoop Yarn Tutorial](https://www.youtube.com/watch?v=6bIF9VwRwE0)
Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,12 @@
1-
# Apache Kafka
1+
# Apache Kafka
2+
3+
Apache Kafka is an open-source stream-processing software platform developed by LinkedIn and donated to the Apache Software Foundation. It is written in Scala and Java and operates based on a message queue, designed to handle real-time data feeds. Kafka functions as a kind of message broker service in between the data producers and the consumers, facilitating efficient transmission of data. It can be viewed as a durable message broker where applications can process and reprocess streamed data. Kafka is a highly scalable and fault-tolerant system which ensures data delivery without loss.
4+
5+
Visit the following resources to learn more:
6+
7+
- [@official@Apache Kafka](https://kafka.apache.org/quickstart)
8+
- [@offical@Apache Kafka Streams](https://docs.confluent.io/platform/current/streams/concepts.html)
9+
- [@offical@Kafka Streams Confluent](https://kafka.apache.org/documentation/streams/)
10+
- [@video@Apache Kafka Fundamentals](https://www.youtube.com/watch?v=B5j3uNBH8X4)
11+
- [@video@Kafka in 100 Seconds](https://www.youtube.com/watch?v=uvb00oaa3k8)
12+
- [@feed@Explore top posts about Kafka](https://app.daily.dev/tags/kafka?ref=roadmapsh)
Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,9 @@
1-
# Apache Spark
1+
# Apache Spark
2+
3+
Apache Spark is an open-source distributed computing system designed for big data processing and analytics. It offers a unified interface for programming entire clusters, enabling efficient handling of large-scale data with built-in support for data parallelism and fault tolerance. Spark excels in processing tasks like batch processing, real-time data streaming, machine learning, and graph processing. It’s known for its speed, ease of use, and ability to process data in-memory, significantly outperforming traditional MapReduce systems. Spark is widely used in big data ecosystems for its scalability and versatility across various data processing tasks.
4+
5+
Visit the following resources to learn more:
6+
7+
- [@official@ApacheSpark](https://spark.apache.org/documentation.html)
8+
- [@article@Spark By Examples](https://sparkbyexamples.com)
9+
- [@feed@Explore top posts about Apache Spark](https://app.daily.dev/tags/spark?ref=roadmapsh)
Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,8 @@
1-
# APIs
1+
# APIs and Data Collection
2+
3+
Application Programming Interfaces, better known as APIs, play a fundamental role in the work of data engineers, particularly in the process of data collection. APIs are sets of protocols, routines, and tools that enable different software applications to communicate with each other. An API allows developers to interact with a service or platform through a defined set of rules and endpoints, enabling data exchange and functionality use without needing to understand the underlying code. In data engineering, APIs are used extensively to collect, exchange, and manipulate data from different sources in a secure and efficient manner.
4+
5+
Visit the following resources to learn more:
6+
7+
- [@article@What is an API?](https://aws.amazon.com/what-is/api/)
8+
- [@article@A Beginner's Guide to APIs](https://www.postman.com/what-is-an-api/)

0 commit comments

Comments
 (0)