Storing APM Synthetic monitor error logs in OCI Logging

OCI(Oracle Cloud Infrastructure) Synthetic Monitoring is used to monitor your application or API endpoint and detect potential availability and performance issues before the end user experiences it.

Intro to APM Synthetic Monitoring:

You will be creating a monitor with any of the below type to monitor the availability and performance of an endpoint.

When you create a monitor with Scripted REST ,whether the run is success or error you will get two logs one for output and another for errors.You can download these logs by navigating to the Monitor History page where the data is stored .

You can create alarm based on the metrics available under the namespace oracle_apm_synthetics.

What if you want to store these logs somewhere or alert based on the data in the logs. In this blog we will see a solution using OCI Function, Alarm and Logging to store these logs .

Solution Design

1. Create a function with the below code as an example .

import oci
import io, json


def get_monitor_logs(apm_domain_id=None, monitor_id=None, vantage_point=None, execution_time=None):
    """Get the APM synthetic monitor error raw logs for a given APM domain,monitor id and vantage point.
    https://docs.oracle.com/en-us/iaas/api/#/en/apm-synthetic-monitoring/20200630/MonitorResult/GetMonitorResult"""
    signer = oci.auth.signers.get_resource_principals_signer()
    apm_synthetics_client = oci.apm_synthetics.ApmSyntheticClient({}, signer=signer)
    try:
        get_monitor_result_response = apm_synthetics_client.get_monitor_result(
            apm_domain_id=apm_domain_id,
            monitor_id=monitor_id,
            vantage_point=vantage_point,
            result_type="log",
            result_content_type="raw",
            execution_time=execution_time)

        error_log = get_monitor_result_response.data.result_data_set[0].string_content
        return error_log
    except Exception as get_log_exception:
        print(get_log_exception, flush=True)


def put_logs(log_id=None, log_content=None, log_ingestion_time=None):
    """Put logs into OCI Logging service.You need to have a log group and log created for this operation.
    https://docs.oracle.com/en-us/iaas/api/#/en/logging-dataplane/20200831/datatypes/PutLogsDetails"""
    signer = oci.auth.signers.get_resource_principals_signer()
    logging_ingestion_client = oci.loggingingestion.LoggingClient({}, signer=signer)
    try:
        logging_ingestion_client.put_logs(
            log_id=log_id,
            put_logs_details=oci.loggingingestion.models.PutLogsDetails(
                specversion="1.0",
                log_entry_batches=[
                    oci.loggingingestion.models.LogEntryBatch(
                        entries=[
                            oci.loggingingestion.models.LogEntry(
                                data=log_content,
                                id="apm_synthetic")],
                        source="APMSynthetic",
                        type="apm_synthetic_error",
                        defaultlogentrytime=log_ingestion_time)]))
    except Exception as put_log_exception:
        print(put_log_exception, flush=True)


def handler(ctx, data: io.BytesIO = None):
    cfg = dict(ctx.Config())
    log_id = cfg.get('log_id')

    try:
        body = json.loads(data.getvalue())
        alarm_type = body.get("type")
        alarm_status = body['alarmMetaData'][0]['status']

        if alarm_type in ["OK_TO_FIRING", 'REPEAT'] and alarm_status == "FIRING":

            resource_id = body["alarmMetaData"][0]["dimensions"][0]["ResourceId"]
            monitor_id = body["alarmMetaData"][0]["dimensions"][0]["MonitorId"]
            vantage_point = body["alarmMetaData"][0]["dimensions"][0]["VantagePoint"]
            execution_time = body.get("timestampEpochMillis")
            log_ingestion_time = body.get("timestamp")

            log_content = get_monitor_logs(apm_domain_id=resource_id, monitor_id=monitor_id,
                                           vantage_point=vantage_point,
                                           execution_time=execution_time)
            if len(log_content) > 0:
                put_logs(log_id=log_id, log_content=log_content, log_ingestion_time=log_ingestion_time)
            else:
                print("No error logs to send", flush=True)

        else:
            print("Alarm Status is not in FIRING state", flush=True)
    except (Exception, ValueError) as ex:
         print(ex, flush=True)

Set the configuration variable named log_id with the value of log OCID.

Create a new custom log in OCI logging using OCI console without agent configuration.

2. Create a notification topic and with the function created before as a subscription.

3. Create an alarm with the desired metrics like availability ,custom metrics etc.. so it will fire when there is an issue in the monitor availability.

For ex: CustomMetric[1m]{CustomMarker = “failed”}.mean() > 0

In the alarm notification section set the topic name to the topic created with function as a subscription previously ,set the REPEAT interval equal to the monitor interval and the message format as Send raw messages.

With this setup you will be receiving the logs in OCI logging when an alarm is in FIRING state and monitor error.log is not empty