Many of my friends and family members, knowing that I work in Data Engineering, often ask me how I gained the skills for my job, how long it took me to develop those skills, and how much it cost. They also inquire about how I landed my first job in this field and how they can start on a similar path. In response, I am creating a guide for those looking to enter the field of Data Engineering to make the journey easier for them.
The career path you choose within the field of data can be influenced by the specific role or responsibilities you want to pursue. In my experience, I have encountered the following types of professionals within the data industry:
Data Scientists: These professionals use statistical and analytical skills to extract insights from data. They often work with large datasets and use tools such as machine learning to make predictions or uncover patterns.
Data Engineers: These professionals are responsible for building and maintaining the infrastructure and systems that allow organizations to store, process, and analyze large amounts of data. They often work with technologies such as Hadoop and Spark.
Data Analysts: These professionals use statistical and analytical skills to extract insights from data and communicate those insights to stakeholders. They often work with tools such as Excel and SQL to manipulate and analyze data.
Business Intelligence (BI) Analysts: These professionals use data to support decision-making within an organization. They often work with BI tools such as Tableau to create dashboards and reports.
Data Architects: These professionals design and build the systems and databases that store and manage an organizationâs data. They often work with technologies such as SQL and NoSQL databases.
Data Managers: These professionals are responsible for the overall management and strategy of an organizationâs data. They often work with teams of data professionals and help to define data governance policies and procedures.
University of Utah: B.S. Information Systems - 2017
I graduated in 2017 with a degree that combined Computer Science and Business from the University of Utah. While I didnât gain a lot of programming knowledge through this program, it did provide me with valuable interpersonal skills that have proved invaluable in my career, such as the ability to work in a team, communicate effectively, plan projects with a group, and meet deadlines. While having this type of degree is not strictly necessary, it can make it easier to secure a job interview for positions in data engineering, data analytics, data science, and ML operations, as it is often listed as a preferred qualification in job postings.
Western Governors University: Masterâs Degree in Data Analytics - 2020
I graduated in 2020 from Western Governors University (WGU) with a degree that included technical training in subjects such as SQL, Python, and machine learning. This program was online and self-paced, and I found it to be very well-structured. It was also much more affordable than a traditional university, but held the same level of accreditation. While having a degree is not a requirement for entering the field of data, it can be helpful during the job interview process and may make it easier to secure a position.
Before 2015, I held jobs in retail, customer service, and the food industry. However, for the purposes of discussing my career path and current position in the field of data, I will begin with the job that is most relevant to my current role.
Microsoft: 2015-2017
At first glance, it may appear that I immediately secured a coding job or internship upon graduating college. However, my first job in the tech industry was actually selling computer hardware at a retail store in the City Creek mall in downtown Salt Lake City. This was my primary job while in college and it provided me with valuable experience and skills, including strong communication skills and the ability to interact naturally with people. By the time I left this position, I was earning approximately $12.50 per hour with full benefits. These skills and experiences have continued to serve me well in my career.
Salt Lake County: 2017-2019
After becoming burnt out by my previous job in retail, I secured a more comfortable position as a Financial Analyst for the local government, earning approximately $19 per hour without benefits. Although I didnât have benefits, it wasnât a major concern as I was still covered by my parentsâ health insurance. My job duties included completing spending reports, updating spreadsheets, and filing paperwork. While it was not the most stimulating work, it allowed me to work part-time while completing my college education.
While working as a Financial Analyst, I was fortunate to be surrounded by supportive colleagues who cared about my career development. I noticed that the organization had a Business Intelligence team and decided to approach them with a request to work on projects with them. They agreed, which allowed me to gain experience writing SQL and building dashboards using Microsoft Power BI.
After graduating college, I received an external offer for a Data Analyst position, but the organization I was currently working for also offered me a full-time role on their Business Intelligence team with the same title. I accepted and stayed with the organization for an additional 1.5 years, during which time I continued to develop my SQL and data analysis skills. My salary increased from $56,000 to $73,000 before I left for Nav.
Nav: 2019-2021
While pursuing my Masterâs degree, I was contacted by a recruiter from a local startup, Nav, on LinkedIn about an opportunity to interview for a Data Engineer position. I agreed and had the interview, but ultimately did not get the job. However, Nav offered me a separate position as a Data Analyst instead. Although my Python skills were not yet developed enough for the Data Engineer role, I accepted the Data Analyst position and focused on improving my skills in SQL and dashboarding (using Tableau instead of Power BI). During this time, I was introduced to a tool called Apache Airflow, which is a Python-based orchestrator used by Nav to sequence SQL queries against a database. This provided me with the opportunity to learn Python from more experienced engineers in a professional setting, as well as to collaborate with multiple engineers on a single project using Git.
Even though I was hired as a Data Analyst, I took on any Data Engineering projects I could within the Apache Airflow system to gain experience writing Python.
After about a year, I received a job offer from 1-800-Contacts as a Data Engineer, the position I had originally applied for at Nav. Without knowing about this offer, Nav offered me a promotion to the same title with a pay increase. I was delighted and decided to remain with the team.
Astronomer: 2021-Present
In the summer of 2021, I was contacted by a recruiter on LinkedIn about an opportunity to work for Astronomer. Although I was happy in my current role at Nav, I felt that Astronomer offered an exceptional opportunity that I couldnât pass up. Astronomer is a company that is commercializing the Airflow open-source project, which we heavily relied on at Nav. Since joining Astronomer, I have been fortunate to work alongside many talented engineers who have a wealth of knowledge and experience. In my role, I help Astronomerâs customers adopt and use Apache Airflow effectively, which involves integrating various data stacks and cloud providers (such as Databricks, AWS, GCP, Azure, BigQuery, Redshift, Snowflake) with Airflow. This has given me extensive hands-on experience with a wide range of data tools commonly used in modern data stacks. Overall, I feel like I have learned more in the past year at Astronomer than I did through all of my higher education and previous job experiences combined.
It is important to consider salary negotiation as you progress in your career. A useful resource for understanding industry salary ranges is levels.fyi, which allows you to view a list of Data Engineering salaries based on years of experience. Keep in mind that salaries at companies like FAANG (Facebook, Apple, Amazon, Netflix, Google) tend to be higher, and salaries may vary based on location. For example, a salary of $200,000 per year in California might be equivalent to a salary of $150,000 per year in Utah.
In the section on my experience, I included salary information for my first few jobs, but not for my more recent positions. While I am not comfortable sharing this information publicly, I am willing to discuss it privately with friends or family members who are negotiating similar job offers.
If I were to start over today and aim for a Data Engineer role, these are the steps I would take to reach my current position:
Step 1: Study SQL
SQL is an essential tool for every data professional (as described above). It is widely used across all types of organizations and is unlikely to become obsolete. In fact, it is likely to be a critical part of any large organization you may work for in the future.
Contrary to what online searches may suggest, you donât need to enroll in a paid course to learn SQL. In fact, I learned more about SQL from YouTube than I did in college (and watching ads is more affordable than tuition). To get started, search for âSQL for Beginnersâ on YouTube and youâll find a wealth of helpful videos. If you find that long videos are overwhelming, try watching 30 minutes to an hour a day and pacing yourself. Itâs also important to follow along with the examples provided in the videos to fully understand the concepts.
Step 2: Study Object Oriented Programming
Learning a programming language is similar to learning a second language in terms of the level of effort required. It is a significant undertaking that requires consistent practice. When choosing a language to learn, it is important to select one and stick with it. If you are new to programming, Python is a good choice. It is easier to read and write code in than Java, although it does compile slower. However, this does not prevent most companies from using it as it is one of the most popular languages today and can be used for a wide range of purposes.
Contrary to what online searches may suggest, you donât need to enroll in a paid course to learn Python. In fact, I learned more about object-oriented programming from YouTube than I did in college (and watching ads is more affordable than tuition). To get started, search for âPython for [insert persona]â (such as âPython for Data Scienceâ or âPython for Data Engineeringâ) on YouTube and youâll find a wealth of helpful videos. If you find that long videos are overwhelming, try watching 30 minutes to an hour a day and pacing yourself. Itâs also important to follow along with the examples provided in the videos to fully understand the concepts.
If you prefer to follow a written guide, I found this book to be very helpful when I was getting started
Step 3: Get Credentials that Prove Your Knowledge
Obtaining additional education or credentials can be a valuable step in your career as a data professional. This could include earning an undergraduate or graduate degree, completing a code bootcamp, or getting certified. When considering this step, it is important to do your research and ensure that the credentials you pursue are seen as credible and respected in the industry. Some certifications or bootcamps may not have a strong reputation and may not be as effective in helping you land a job. On the other hand, certain certifications from reputable companies may be highly regarded and help you stand out in your job search. Here are a couple of examples of what I would consider worthwhile certifications:
The goal of acquiring additional education or credentials is to increase your chances of being invited for an interview for your desired role. Therefore, it is important to focus on the skills that are required or preferred for your targeted job title and tailor your learning accordingly. By doing this, you can better position yourself to meet the qualifications for the job and stand out as a strong candidate.
Step 4: Apply for Internships (Optional)
If I were to do things differently, I would try to spend less time working in the retail and service industries during college. While these jobs can be necessary to make ends meet, it would have been more beneficial for me to focus on gaining coding and technical skills as quickly as possible. One way to do this is through internship programs offered by companies such as FAANG or large tech firms, although these can be competitive. If you are unable to secure an internship with these companies, consider looking for opportunities with local startups or other companies. These organizations may be more willing to provide opportunities for students to learn and develop technical skills.
Step 5: Cater your Resume & LinkedIn
To increase your chances of success in the data industry, it is important to establish yourself as a credible and skilled professional on LinkedIn. One way to do this is by branding yourself as the type of data professional you aspire to be. This can include making posts about your progress and accomplishments, listing any certifications or qualifications you have obtained, and connecting with others who are on similar learning paths. Additionally, be sure to keep your work experience and other fields on your LinkedIn profile up-to-date to showcase your skills and capabilities to potential employers. By taking these steps, you can demonstrate your commitment to the field and increase your visibility to potential employers.
It is important to have a well-organized, up-to-date resume that clearly showcases your skills and accomplishments. A one-page resume is generally sufficient for most data professionals and allows you to highlight your relevant experience and qualifications in a concise manner. If you are in need of a resume template, you may find it helpful to use mine as a starting point. Having your resume reviewed by career advisors or other professionals can also be helpful in ensuring that it is effective in highlighting your strengths and qualifications.
Step 6: Practice Coding Challenges
One of the most effective ways to prepare for the technical portion of a data-related job interview is to practice solving puzzles and challenges related to the skills and technologies you will be expected to know. Websites like HackerRank offer a variety of programming exercises and challenges that can help you hone your skills and build confidence in your abilities. By regularly practicing and solving these types of problems, you can be better prepared to demonstrate your technical skills and knowledge during an interview.
Explore the seamless integration of secrets management in Apache Airflow, allowing you to customize connections for different development environments. Illustrated through the perspective of our Data Engineer hero, Bob.
Explore the art of transforming JSON data in Amazon Redshift databases, mastering elegant techniques to split and extract data from nested structures. Uncover time-saving strategies for managing one-to-many relationships and enhancing data manipulation workflows.
Discover the distinctions between Data Analyst, Data Engineer, and Data Scientist roles and learn the essential skills required to transition from one to the other.
