Aspera-pot
Aspera faspio gateway with IBM MQ Managed File Transfer (MQMFT)
Return to Aspera faspio labs page
Table of Contents
- 1. Overview
- 2. Prepare the lab environment
- 3. MQ Managed File Transfer Setup
- 4. MQ Channels Verify
- 5. Testing TCP File Transfer
- 6. Switch MQ Channels to FASP
- 7. Testing FASP Transfers
- 8. Summary
1. Overview
In this laboratory, you will explore the benefits of employing the IBM Aspera faspio Gateway alongside IBM MQ Managed File Transfer Queue Managers and its Agents that are situated in two remote regions.
What is Aspera faspio Gateway?
IBM Aspera faspio Gateway is a software component that accelerates TCP-based data transfers between systems, leveraging the Aspera FASP protocol for high-speed, reliable, and secure data movement over any distance. It acts as a bridge, enhancing data transfer performance, particularly over networks with high latency or packet loss.
Architecture Diagram
The IBM MQ Managed File Transfer Coordination, Command and Managed File Transfer Agent(s) Queue Managers between AMERICAS, and EUROPE are linked via a set of Sender/Receiver TCP channels.
Then, we will leverage IBM Aspera faspio bridge between the IBM MQ Sender/Receiver Channels to speed up file transfers between the two Queue Managers.
Environment
You will make use of two IBM Techzone environments situated in the AMERICAS and EUROPE regions.
Testing
You will be testing two usecases.
a) Send 1GB file over TCP Channel
b) Send 1GB file over FASP Channel
Then, compare the results TCP vs FASP.
Note:
faspio lab1 is a pre-requirement. lab1 will takes care of faspio gateway setup, creation of the Agent Queue Managers (QMWDCD01, QMLDND01) between the regions.
2. Prepare the lab environment
Capture IP address of AMERICAS and EUROPE environments.
Save the IP Address(s) to a Notepad. You will need these to setup faspio gateway on both the regions.
3. MQ Managed File Transfer Queue Manager Setup
In this step, you will setup IBM MQ Queue Mansgers for Managed File Transfer Queue Manager for Coordination & Command, Agent Queue Manager in AMERICAS, and Agent Queue Manager in EUROPE.
3.1 AMERICAS Environment Setup
You are tasked with establishing a Queue Manager named QMMFTD01. QMMFTD01 will be used as a MQ Manaaged File Transfer Coordination Queue Manager. QMWDCD01 (where WDC denotes Washington DC), which you have created in the previous lab will be used as Managed File Transfer Agent Queue Manager.
Open the Windows VM.
Login to the windows VM as techzone / IBMDem0s.
Open Putty Program, and aspera3 virtual machine.
cd ~/faspio
./setup-americas-europe-qmgr-mqmft.sh <EUROPE-SERVER-IP-ADDRESS>
Review the results.
The script will create QMMFTD01, and an Managed File Transfer Agent called AGTWDCD01.
Note: Due to the time limitations of the workshop, we streamlined the lab process and developed scripts for establishing the MQ Queue Managers and their connectivity. Kindly review the scripts if time allows.
3.2 EUROPE Environment
QMLDND01 (where LDN denotes London), which you have created in the previous lab will be used as Managed File Transfer Agent Queue Manager.
Open the Windows VM.
Login to the windows VM as techzone / IBMDem0s.
Open Putty Program, and aspera3 virtual machine.
cd ~/faspio
./setup-europe-americas-qmgr-mqmft.sh <AMERICAS-SERVER-IP-ADDRESS>
Review the results.
4. Verify MQ Channels & Agents
Now, lets verify the MQ Channels are in RUNNING state.
4.1 AMERICAS Environment - MQ Channel Verify
Run the below command from aspera3 terminal window.
echo "dis chstatus(*)" | runmqsc QMWDCD01
4.2 AMERICAS Environment - MQ Managed File Transfer Agents Verify
fteListAgents
5. Testing TCP File Transfer
It is now the moment to evaluate our efforts.
5.1 AMERICAS Environment - MQ Explorer
From the Windows Virtual Machine, open MQ Explorer that can be launched from the Windows Desktop or the taskbar.
Make sure you can connect to all three Queue Managers. if you see below dialog, just select Yes.
You may need to re-add QMLDND01 with correct ROUTER-WAN-IP Address.
Open Putty program, and launch “aspera3” virtual machine.
Logon as ibmuser/engageibm.
5.2 AMERICAS Environment - Transfer 1GB file
Run the following commands from “aspera3” command line as you did above.
dd if=/dev/zero of=/tmp/wdc-1gb-tcp.txt bs=1024 count=1048576
fteCreateTransfer -rt -1 -jn “test-wdc-ldn-1” -sa AGTWDCD01 -sm QMWDCD01 -da AGTLDND01 -dm QMLDND01 -sd delete -dd “/tmp/” “/tmp/wdc-1gb-tcp.txt “ -de overwrite
Notice the transfer speed is about 12MB/s.
When the transfer is completed, Notice that the elapsed time to transfer 1GB file between AMERICAS and EUROPE is about 1 minute and 17 seconds.
Now, let’s transfer a 10GB file.
dd if=/dev/zero of=/tmp/wdc-10gb-tcp.txt bs=1024 count=10485760
fteCreateTransfer -rt -1 -jn “test-wdc-ldn-2” -sa AGTWDCD01 -sm QMWDCD01 -da AGTLDND01 -dm QMLDND01 -sd delete -dd “/tmp/” “/tmp/wdc-10gb-tcp.txt” -de overwrite
Notice that the transfer speed is about 13-14MB/sec.
Notice that it took 13 minutes and 40 seconds to transfer 10GB file.
6. FASP Switch - Switch MQ Channels to FASP
Now, lets leverage faspio gateway between the MQ Queue Managers. Let’s modify the MQ Agent Channels to use FASP Gateway.
6.1 AMERICAS Environment - Switch MQ Channels to to FASP
Let’s configure MQ Channels to route traffic to the faspio Gateway.
Open Putty program, and launch aspera3 virtual machine.
Logon as ibmuser/engageibm. Run the following commands.
cd ~/faspio
./switch-americas-europe-mqchannel-tcp-to-fasp.sh
6.2 EUROPE Environment - Restart MQ Receiver Channel
Run the below command to restart WDC.LDN channel.
echo "stop channel(wdc.ldn)" | runmqsc QMLDND01
echo "dis chstatus(wdc.ldn)" | runmqsc QMLDND01
Make sure the channel is stopped. <br>
echo "start channel(wdc.ldn)" | runmqsc QMLDND01
7. Testing FASP Transfers
AMERICAS Environment, open Putty program, and launch aspera3 virtual machine.
Logon as ibmuser/engageibm.
7.1 AMERICAS Environment - Transfer 1GB file
Run the following commands.
dd if=/dev/zero of=/tmp/wdc-1gb-fasp.txt bs=1024 count=1048576
fteCreateTransfer -rt -1 -jn “test-wdc-ldn-3” -sa AGTWDCD01 -sm QMWDCD01 -da AGTLDND01 -dm QMLDND01 -sd delete -dd “/tmp/” “/tmp/wdc-1gb-fasp.txt “ -de overwrite
Check “Transfer Log” from the MQExplorer. Notice that transfer speed is now at 45MB/s
Check the elapsed time with FASP is now at 33seconds.
Now, let’s transfer a 10GB file. Run the following commands
dd if=/dev/zero of=/tmp/wdc-10gb-fasp.txt bs=1024 count=10485760
fteCreateTransfer -rt -1 -jn “test-wdc-ldn-4” -sa AGTWDCD01 -sm QMWDCD01 -da AGTLDND01 -dm QMLDND01 -sd delete -dd “/tmp/” “/tmp/wdc-10gb-fasp.txt” -de overwrite
Notice that the transfer speed is running about 45MB/s.
Notice that the elapsed tiem is about 3 minutes and 44 seconds.
8. Summary
Now compare the difference between TCP Transfer, and FASPIO gateway transfer of 1GB file. You should see 130% more speed on 1GB, and 266^ more with 10GB file transfer. As the file size, and distance increase, FASP throughputs increase as well.
