How can you monitor and manage the activities of an Azure architecture with custom services? In this article, we will present two possible custom additional services that enable the automatic sending of alert emails and a monitoring report of Azure activities.
An Azure architecture, especially if complex, requires management and monitoring support.
While Azure provides monitoring services, these are not always user-friendly for less experienced users and may not always be accessible promptly via the Azure portal to detect critical situations. In this article, we present two enhancements that can simplify and facilitate the daily activities necessary for optimal maintenance of the structure, operations, and performance, and enable timely management of crisis situations.
The first option we will discuss involves implementing an automated email notification system for activities performed. Another potential development could be implementing a monitoring report for the activities performed using appropriate log configuration tables. Compared to the Azure Data Factory monitor, this approach has the advantage of being more accessible to non-technical users and is also available on smartphones.
Automated Customizable Email Sending via the Free Azure Logic App Service
Among the many services offered by Azure, Azure Logic App stands out as an essential and free resource for subscription holders. This service allows, among other things, the implementation of an automatic notification system to monitor and manage critical activities. This implementation is particularly beneficial for enhancing the architecture of a Data Platform system, enabling a timely response in critical situations, such as missed data updates.
Azure Logic App is an advanced cloud platform that enables the creation and execution of automated workflows without the need for coding, following the code-less principle. Through an intuitive visual designer and leveraging a wide range of predefined actions, it is possible to quickly develop flows that integrate and manage applications, data, services, and systems coherently.
Notifications are a crucial element in any business environment. They are the bridge between data and decisions, ensuring that stakeholders receive the information they need at the right time. Here’s how Azure simplifies this process by involving three main components: Azure Data Factory, Azure Automation, and the Azure Logic App service.
Using a service email account, such as Office 365, allows for the automation of email sending in various situations without needing to create ad-hoc flows for each specific case. This strategic approach enables efficient and centralized management of notifications, contributing to the coherence and effectiveness of the overall system.
But how does it work in practice? To begin, you need to set up a configuration table containing information about the reference Data Factory environment. This information includes the name of the Azure Data Factory resource, the pipeline name, recipients, and more.
Once configured, Azure Data Factory takes care of the rest. A pipeline is created that links to the configuration table. Managing configuration in a table allows for full customization, such as body, sender, subject, offering a detailed level adapted to the case. For example, you can use this pipeline in a context where you manage the update of different tables. If an update fails, the details can be captured and used to enrich the content of the emails. It’s an efficient solution for providing case-specific information. Each record extracted from the pipeline will generate an email notification.
To make everything operational, you need to insert the Azure Logic Apps URL for automatic email sending. As seen in the image below, it is possible to include extra controls to ensure security. Only authorized calls can actually send the emails. This is essentially a security mechanism to reduce the risk of a “denial of service” (DoS) attack, where a third party deliberately damages other services by overwhelming them with thousands of calls. In other words, consumption is related only to the actual sending of emails, not to the Logic Apps calls.
In summary, Azure simplifies the process of automatic notifications with an intuitive configuration and secure control; it offers an effective solution to ensure that the right people receive the necessary information to focus on what matters most for the business.
Power BI Reporting for Monitoring Azure Activities
Another interesting development is implementing reporting to monitor performed activities, using the log configuration table. The log table contains information related to the activities performed, such as the name of the executed pipeline, the table to be updated and its model, the process phase, and execution time, allowing us to save useful data to reconstruct events and calculate relevant statistics.
Unlike the standard monitoring of Azure Data Factory, this solution has the advantage of being easily accessible even to those with no advanced technical skills and is also accessible from mobile devices like smartphones through the Power BI application.
The overview of this reporting focuses on the log table, providing a historical view of performed activities.
In the image below, you can observe a table detailing all performed loads, providing crucial information related to the pipeline enabling the update, the reference table, and its model. The table includes details on the update status (success, failure, or still in progress), start and end dates and times, along with some relevant statistics.
This visualization offers an immediate overview of performed activities, enabling efficient monitoring of loading operations. Details about the pipeline, table, and model provide key context. Information about the status offers a quick assessment of the success or failure of the update, while start and end timestamps provide essential temporal perspective.
The statistics of interest in the table may vary, but here we can see details on loading duration, the wait time before the update starts, as well as the number of updated partitions, degree of parallelism, and priority assigned to the table. This detailed representation aims to provide a comprehensive picture of loading activities, facilitating an accurate and timely assessment of the overall system performance.
At a more detailed level, fundamental synthetic parameters for evaluating system performance are examined. Process durations are analyzed compared to the average duration, providing a clear indication of temporal variations in update activities. Additionally, the queue wait time is assessed, providing crucial information on the smoothness of operations and their overall temporal efficiency.
Special attention is given to the trend of updated partitions relative to the average value. This analysis provides a dynamic view of variations in partition update activities over time. Users can thus evaluate the impact of changes over time and understand how many partitions are actually involved in update operations.
Partition efficiency is another key aspect considered in this analysis. It evaluates the number of partition data loaded that actually needed to be updated. This provides a direct measure of the effectiveness of update operations, helping to identify potential areas for optimization and improvement.
We realize Business Intelligence & Advanced Analytics solutions to transform simple data into information of freat strategic value.
