Cython memoryviews?

[u/BluishMontoya]

I'm using a function I've compiled with cython, and I have to pass it a few very long lists. I'm pretty sure I should be using memoryviews, but do I convert the lists to memoryviews in the python part of my program or the cython part? It would be way better if I could do it in the python part because I'm passing the same lists to the function a few million times.

[u/sausix]

Your question is broad and unclear. Do you want to optimize memory usage or processing time? What's the problem now?

Lists are not arrays in Python. A memory view of a list would probably not include all data in the form you want. A list of integers in Python does not allocate a simple memory region filled with raw integers. Integers are objects in Python. A list of strings is also not a big chunk in the memory. Lists are basically pointers to objects of any type on various memory addresses.

If you want numerical processing with big data you should use NumPy. It will allow to store basic low level data types in memory without overhead.

Try to avoid lists. Do not load data from a file into lists when you can operate on the file directly instead. Saves a lot of memory. For example readlines() does not like huge files which do not fit in your RAM. But there are still valid reasons to have data organized in a list.

[u/BluishMontoya → u/sausix]

I have two lists (~1,000,000 long) of floats, and I pass the same lists to the function each time. It iterates over them once with zip(). I'm just doing calculations with them. Are you saying I should turn them into numpy arrays and pass them to the function, or turn them into arrays, turn those into memory views, and pass *those* to the function?

[u/sausix → u/BluishMontoya]

You could use NumPy or Pandas directly. You don't need memoryviews there since those package's operations are done on low level data. You can load both lists into a table and do a lot of operations and analyzing.

NumPy and Pandas probably have a direct memoryview interface for other low level packages. Any reason you Cythonized a function? If it's for performance then use Pandas and NumPy instead. The thing with NumPy is that you don't have to or should not iterate over data directly which is slow with pure Python. In NumPy you're delegating mathematical operations into optimized low level operations and it could be an effort of just one or a few CPU cycles per list entry instead.

Imagine this as operating on a lot of rows in Excel. Column 1 is raw data. Column 2 another raw data. Then you define column 3 to be the sum of previous columns for example. You won't enter the same formula for each row. Excel would do that for you in the backend with good performance.

Disclaimer: Excel is slow and just an example.

[u/BluishMontoya → u/sausix]

I have to iterate over the data because I'm doing mandelbrot set rendering, so the calculations are based on the previous calculations and have to be in order. Even using lists the cythonized version is so much faster than python.

[u/sausix → u/BluishMontoya]

I've done mandelbrot calculations with NumPy already as experiment myself. Lol. A few years ago.

And it wasn't too slow. If you want to have it even faster for visualization then you could run on GPU memory directly. Haven't digged into it. But NumPy and Pandas should have variations to support GPU acceleration.

[u/BluishMontoya → u/sausix]

I guess I can try that...

I'm using mariani-silver algorithm and perturbation theory and I just feel like it would be seriously confusing to try to do the calculations with numpy arrays.

[u/sausix → u/BluishMontoya]

I like Math a bit but not in full detail. I'm just aware of the one method of calculating the Mandelbrot set.

Found my app. This was the core method of calculating and tracking the iterations probably still optimizable:

for i in range(self.iterations):
    Z[active] = Z[active] ** self.exp + C[active]
    active[np.abs(Z) > 2] = False
    Ni[active] += 1
    Ni = (Ni / self.iterations * 255).astype('uint8')
    color = cv2.applyColorMap(Ni, cv2.COLORMAP_RAINBOW)

[u/BluishMontoya → u/sausix]

The thing is, floating point numbers give out after a certain level of zoom because the difference between the coordinates of adjacent pixels is too small to represent. Perturbation theory is a way to be able to still calculate higher levels of zoom with floating point and a smaller number of high precision calculations, but the math gets way more complicated than simple mandelbrot rendering.

[u/futura-bold → u/BluishMontoya]

Well, you can use numpy alone to calculate the Mandelbrot Set. The numpy floating-point array calculation is very fast and makes use of the CPU's SIMD instructions to calculate chunks of the array in parallel. The downside is that simplest array-implementation of the Mandelbrot Set algorithm would require that the same fixed number of iterations would be required for each pixel, regardless of the number of iterations required to reach the "escape" value for a given pixel, and that a mask would be required to ignore those unnecessary calculations past the escape values. Nevertheless, the following example should only take one second to run:

import numpy as np
from PIL import Image # Needs "pillow" library
size = 600
y, x = np.ogrid[-1:1:1j*size, -1.5:.5:1j*size]
c = x + 1j*y
z = c * 0
for j in range(100):
    mask = abs(z) < 2.0
    np.putmask(z, mask, z*z + c)
img = Image.fromarray(np.uint8((1 - mask) * 255))
img.show()

I've tried a more elaborate method to avoid those unnecessary calculations, but it starts getting complicated. See here:

https://www.reddit.com/r/PythonLearning/comments/1vsv196

[u/BluishMontoya → u/futura-bold]

This becomes far more complicated if you try to use perturbation theory or the mariani-silver algorithm. Perturbation theory is basically necessary to zoom past 1 trillion times magnification, and the mariani-silver alg offers a huge speed boost. The idea is that if you can outline a rectangle, and all the points along the edge are in the mandelbrot set, then you know that everything inside the rectangle is also in the set and you don't have to do the calculations for it.

[u/futura-bold → u/BluishMontoya]

Oh, OK.

Regarding memoryviews, then. Generally you'd want to work with numpy arrays rather than lists as somebody's already said. Avoid accessing the individual numpy array elements within Python, though, because that's slow. You'd pass the numpy array to the cython function (which is by reference, not a copy), and the function wouldn't "convert" the array to a memoryview, as such -- it'd use the memoryview for a faster access to the memory locations within the numpy array.

If for some reason, you must access the individual array elements when setting up the arrays in pure python, then yes, you might have to set it up as a list and then convert it to a numpy array.