Low-Level PyMAD-NG

Sending and Receiving Multiple types

To start, the file ex-LowLevel/ex-send-multypes.py imports all the necessary modules, creates a large numpy array, setups the mad object to communicate with MAD-NG, and then creates a string to send to MAD.

The string starting on line 9 below creates a 1000x1000 matrix within MAD-NG and generates the matrix into a sequence, see MAD-NG documentation on mat:seq for more details. Then on line 11, MAD-NG is asked to send the matrix back.

 1from pymadng import MAD
 2import numpy as np
 3import time
 4
 5arr0 = np.zeros((10000, 1000)) + 1j  # 2*10000*1000*8 -> 160 MB
 6
 7mad = MAD()
 8
 9## Create a matrix in MAD and send it to Python
10mad.send("""
11    local m1 = MAD.matrix(1000, 1000):seq()
12    py:send(m1)
13    """)

The next section creates a complex matrix in MAD-NG named cm1, then creates and asks MAD-NG to send back to python multiple variations of this complex matrix with calculations performed on them.

 1## Create a complex matrix in MAD
 2mad.send("cm1 = (MAD.cmatrix(10000, 1000) + 1i)")
 3
 4# Create a string the manipulates the complex matrix in MAD and sends the result it to Python
 5cmatrixString = """
 6    {0} = cm1 {1} {2}
 7    py:send({0})"""
 8
 9mad.send(cmatrixString.format("cm4", "*", 1)) ## Set cm4 to cm1 * 1 and send it to Python
10mad.send(cmatrixString.format("cm1", "*", 2)) ## Set cm1 to cm1 * 2 and send it to Python
11mad.send(cmatrixString.format("cm2", "*", 2)) ## Set cm2 to cm1 * 2 and send it to Python
12mad.send(cmatrixString.format("cm3", "/", 3)) ## Set cm3 to cm1 / 3 and send it to Python

Then the same is done for a single vector of length 45.

1## Create a vector in MAD and send it to Python
2mad.send("""
3    local v1 = (MAD.vector(45):seq()*2 + 1)/3
4    py:send(v1)
5    """)

We then receive all of the variables in the same order they were sent, time the transfer and check that the correct calculations we performed. If everything goes well, a time will be followed by four Trues

 1start_time = time.time() # Start timer
 2
 3# Receive the matrices and vectors
 4m1 = mad.recv() 
 5cm4 = mad.recv()
 6cm1 = mad.recv()
 7cm2 = mad.recv()
 8cm3 = mad.recv()
 9v1 = mad.recv()
10
11print(time.time() - start_time) # Print time
12
13# Check if the matrices have been correctly sent
14print(np.all(cm1 == arr0*2)) 
15print(np.all(cm2 == arr0*2*2))
16print(np.all(cm3 == arr0*2/3))
17print(np.all(cm4 == arr0))

Important

The examples above break what we describe as sequencing, therefore, if you are not careful with the procedure, you may end up with a deadlock. This is because both MAD-NG and python can wait for the other to receive the data that is being sent. See below for more details.

The rest of the ex-send-multypes.py file shows and tests sending and receiving some of the available types.

Creating Deadlocks

To cause a deadlock, the easiest way is to send and ask for lots of large data without handling the data in python. For example, if we were to ask for a large matrix, and then send another large matrix, without handling the first matrix, then MAD-NG will be waiting for python to receive the first matrix, while python is waiting for MAD-NG to send the second matrix (see below).

# Create an array much bigger than 65 kB buffer
arr0 = np.zeros((10000, 1000)) + 1j
# Ask MAD-NG to receive data from python
mad.send('arr = py:recv()')
# Send data to MAD-NG
mad.send(arr0)

# Ask MAD-NG to send data back to python.
# Since this fills the buffer, it will hang until python receives the data
mad.send('py:send(arr)')     # Danger starts here, as MAD-NG is hanging, waiting for python to receive the data
arr2 = arr0 * 2              # Manipulate data in python

# Ask MAD-NG to receive more data from python
# (this will not be executed until python receives the data from MAD-NG)
mad.send('arr2 = py:recv()')

# DEADLOCK! You have now filled the buffer on both sides,
# and both are waiting for the other to receive the data, so nothing happens
mad.send(arr2)

Sending and Receiving Large Datasets

The file ex-LowLevel/ex-send-recv.py shows sending 960 MB arrays and then receiving manipulated versions of these arrays and verifies the manipulations. This example uses very large datasets, that, if not careful, can cause a deadlock.

On the first line below, we ask MAD-NG to receive some data from python, and then on the second line, we send the data, if we forget to do this, MAD-NG will error or receive gobbledygook as the contents of the matrix, as it tries to interpret the string on line 4 as a matrix.

1mad.send("cm1 = py:recv()") # Tell MAD to receive something 
2mad.send(arr0)              # Send the data to MAD
3
4mad.send("cm2 = py:recv()") 
5mad.send(arr0)

Important

Therefore from this and the previous section, the general rules are:

  • If you ask MAD-NG to receive data from python, you must immediately send the data to MAD-NG

  • If you ask MAD-NG to send data to python, you should immediately receive the data from MAD-NG

Sending and Receiving Scripts

Finally we test the receiving scripts and execution, where we find the first script sent to MAD-NG must be executed twice by python, as the first command sends a command back to MAD-NG, which sends a command to be executed by python.

Then the second script explicitly sent to MAD-NG, extends the first script, so that python is required to execute the script 3 times.

# Send a string to MAD that will send a string back to Python
mad.send("""
    py:send([=[
mad.send("py:send([[print('pymad success')]])") # Needs to lack indent for python
    ]=])""")
mad.recv_and_exec()
print("On second read:")
mad.recv_and_exec()

mad.send("""
    py:send([==[
mad.send('''py:send([=[
mad.send("py:send([[print('Another pymad success') #This MAD string is executed in Python]])")
    ]=])''')
    ]==])
    """)
mad.recv_and_exec()
mad.recv_and_exec()
print("On third read:")
mad.recv_and_exec()

On success of both scripts, the print command sent by MAD-NG will only be executed after the explicit print commands, visible in python

Local and Global Scopes

If you are used to coding in MAD-NG, you will be familiar with the use of the statement local, which allows you to define variables in the local scope. This is done because as you increase the number of global variables, the slower the code will run, and because otherwise you will get a warning from MAD-NG. In PyMAD-NG, it is not entirely necessary to use, but it is recommended in order to ensure scope management, however you will not get a warning if you do not use it. This is because every use of the mad.send() function is an separate scope, so if a variable is made local in one scope, it will not be available in another scope. If the local is ommitted, then the variable will exist within the scope of the MAD instance, and will be available to all subsequent calls to mad.send(). To see this in action, see below:

mad.send("""
a = 10
local b = 20
print(a+b) -- 30
""")
mad.send("print(a+(b or 5))") # b is not defined, so it becomes a + 5 = 15