11/18/2023 0 Comments Aws mwaa tutorialIf the answer to these is yes, then it is likely that using a managed service may frustrate you. It is important to assess a true like for like, and we often see just the actual compute and storage resources being compared without all the additional things that you need to make this available. Total Cost Ownership One thing to consider when assessing managed vs self-managed is the cost of the managed service against the total costs of you having to do the same thing. Whether you have the need to run workflows that use more resources that managed services provide (for example, need significant compute).Whether have the need to have the very latest versions or features of Apache Airflow.Whether you need the increase level of access, a greater level of control of the configuration of Apache Airflow.Typically this choice depends on a number of factors based on your particular business or use case. One of the first decisions you have to make is whether you want to use a managed versus self-managed Apache Airflow environment. How do you enable these for use within your Apache Airflow environments? New Python libraries are released on a frequent basis, new data tools are also constantly changing.Deploying workflows to production can break Apache Airflow, so how do you minimise this?.Ensuring that you bake security from the beginning, how can you separate concerns and make sure that you minimise the number of secrets that developers need access to?.How do you ensure that you can provide the best developer experience and to help minimise the issues with deploying workflows to production?.How quickly are you able to release updates and changes to support their need? There is constant innovation within the Apache Airflow community, and your data engineers will want to quickly take advantage of the latest updates.Do you have the skills or the desire to want to manage this? Apache Airflow is a complex technology to manage, with lots of moving parts.It is worth spending some time to understand some of the challenges scaling Apache Airflow. For others, the availability of managed services, such as Managed Workflows for Apache Airflow (MWAA), has helped reduce the complexity and operation burden for running Apache Airflow, and opened this up for more Builders to start using it. For some, self managing Apache Airflow is the right path to take. Like many open source technologies, there are many ways in which you can configure and deploy Apache Airflow.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |