MQ-Advanced-pot

MQ Open telemetry Tracing with Jaeger

1. Overview

In this lab, we will explore how to capture and send MQ Open Telemetry traces to Jaeger user interface. Jaeger is an open-source distributed tracing tool to monitor and troubleshoot MQ transactions.

Lab environment:
Platform: RHEL Linux VM
Jaeger Runtime: Podman

2. Installing Jaeger

Let’s install & run Jaeger as a podman container. Assuming the RHEL VM already has podman installed.

# install crun if not already installed. crun is a fast and lightweight OCI (open container initiative) runtime.

sudo yum install crun

podman run -d --restart unless-stopped --name jaeger -p 16686:16686 -p 4317:4317 quay.io/jaegertracing/all-in-one:latest

# Check if Jaeger container is running
podman ps

Open Jaeger console http://localhost:16686

3. Install MQ OTEL Tracing libraries into /var/mqm/exits64

Download MQ OTEL Tracing libraries from here</b>

Use unzip commands to extract files.

sudo unzip ibm-mq-tracing-exits_xLinux_64bit.zip -d /var/mqm/exits64

sudo chown mqm:mqm /var/mqm/exits64

4. Update mqs.ini

IBM MQ Tracing uses an IBM MQ user exit to trace IBM MQ messages.

Append below stanza to /var/mqm/mqs.ini file.

sudo vi /var/mqm/mqs.ini

ApiExitCommon:
  Sequence=100
  Function=EntryPoint
  Module=/var/mqm/exits64/mqtracingexit
  Name=TracingApiExit

5. Create Queue Manager

# Create a queue manager
crtmqm OTEL

# Start the queue manager
strmqm OTEL

# Create a test queue
echo "define qlocal(AA)" | runmqsc OTEL

6. Testing - Exploring Tracing

Lets use amqsput, and amqsget sample programs to put & get messages to AA queue.

*** Open first Terminal window.

# Put one message
/opt/mqm/samp/bin/amqsput AA OTEL

*** Open second terminal window.

# Get message
/opt/mqm/samp/bin/amqsget AA OTEL

Now, from the first terminal window enter a message.

Open Jaeger console in browser and check the generated traces.
http://localhost:16686

Select OTEL Queue Manager from the Service dropdown field.

Click by the ‘2 spans’ and some more details are presented.

Notice the PUT part runs from 0 to 499 microseconds and the GET does nothing ( black line) until 499 microseconds.

The trace is indicating that in this scenario (with a ‘getting’ program in place to immediately process anything that arrives on queue AA) the time taken to put the message on the queue is 499 microseconds and the time taken to retrieve the message from the queue was about .500 microseconds ( the difference between 499 microseconds and 999 microseconds).

You can click on the individual lines in the Jaeger UI to get more details

7. Exploring trace logs

When the tracing exits and configuration file were installed, the file mqtracingexit.conf set a value of LOG_LEVEL=”info”. This is requesting a basic log file.

Check /tmp a notice a new folder has been created /tmp/ibmmqtrace, and contains some mqExit.xxx.log files. Look at one of the log files.

Let’s change the LOG_LEVEL to ‘debug’.

Update /var/mqm/exits64/mqtracingexit.conf file.

sudo vi /var/mqm/exits64/mqtracingexit.conf

LOG_LEVEL='debug'

Restart the Queue Manager.

endmqm OTEL
strmqm OTEL

Put a test message.

/opt/mqm/samp/amqsput AA OTEL 

Now check the logs under /tmp/ibmmqtrace folder. Check the latest log file. You should see debug traces getting created like below.

Scroll down until you see mqttax.spanctx which shows the span identifier.

And notice that this matches the field SpanID in the Jaeger UI display.

At this point , it should be clear that the MQ tracing exits have generated a spanid on both the MQPUT and the MQGET calls and then passed this information to a collector (Jaeger).

Although the MQPUT and MQGET call each had their own spanid, the Jaeger UI showed them as linked. The reason is that when the MQGET call created a new span id it was also aware of the original (MQPUT) spanid and used the original span id as a ‘parent’ when passing information to the Open Telemerty Jaeger collector.

You maybe asking the question “how does the MQGET know the spanid generated by the MQPUT?”.

[ The AMQSPUT and AMQSGET are in different programs running in different terminals].

The span id of the MQPUT call was included in the MQ message using a property folder. This is revealed if you browse the message on queue AA. This screen shot shows an RFH2 header is used to pass the original span id from the MQPUT in the spanctx> section ( and remember that earlier we saw mqttax.spanctx was mentioned in the /tmp/ibmmqtrace log files).

That concludes the MQ Open Telemetry tracing lab. The OTEL traces can be sent to Instana or any other Tracing platform that can accept Open Telemetry traces to monitor MQ transactions.

8. References

MQ Open Telemetry Integration
https://www.ibm.com/docs/en/ibm-mq/9.4?topic=network-opentelemetry-integration

Further information about the exit and options of the values in the configuration file at.
https://www.ibm.com/docs/en/instana-observability/current?topic=mq-tracing#ibm-mq-tracing

End