Python3, multiprocessing shared memory for modules?

[Poet129]

I have a python program that creates os.cpu_count() subprocesses, and they all have to import a large module. Is there a way to only import the module once into memory and share it across the subprocesses? Source

[Poet129]

3 hours ago, gabrielcarvfer said:

Not that I'm aware. You could try using the shared_memory construct, but you would need to parse the module and it would spend a lot more memory anyways. Just use less processes and it should be fine.

I think I got somewhere...

from multiprocessing.shared_memory import SharedMemory
from torch import manual_seed, randint
w = bytes(str(manual_seed), "utf-8")
x = len(w)
shm_a = SharedMemory(create=True, size=x)
buffer_a = shm_a.buf
buffer_a[:x] = w
y = bytes(str(randint), "utf-8")
z = len(y)
shm_b = SharedMemory(create=True, size=z)
buffer_b = shm_b.buf
buffer_b[:z] = y
seed = str(bytes(buffer_a[:x]))
seed = seed.replace("b'", "")
seed = seed.replace("'", "")
#seed.__class__ =
print(manual_seed)
print(manual_seed.__class__)
print(seed)
print(seed.__class__)
ranint = str(bytes(buffer_b[:z]))
ranint = ranint.replace("b'", "")
ranint = ranint.replace("'", "")
#ranint.__class__ =
print(randint)
print(randint.__class__)
print(ranint)
print(ranint.__class__)

Using that code I get this...

<function manual_seed at 0x000002BF9FAAB3A0>
<class 'function'>
<function manual_seed at 0x000002BF9FAAB3A0>
<class 'str'>
<built-in method randint of type object at 0x00007FF9C9AF2680>
<class 'builtin_function_or_method'>
<built-in method randint of type object at 0x00007FF9C9AF2680>
<class 'str'>

If I changed (seed and ranint)'s classes back to what they were before would this work? Reason I'm asking is I can't figure out what to put after the following code:

seed.__class__ = 
ranint.__class__ =

What would I set those to, to make them the same as before running though shared memory? Yes, I'm aware I haven't created any subprocesses to test this completely yet, but I needed this to work first. Any help is appreciated.

[Poet129]

@gabrielcarvfer

I ended up with this code for testing:

def memorylocalseed(manual_seed):
    openfile = open("Results.txt", "a")
    openfile.write(str(manual_seed) + "\n")
    openfile.close()
def memorylocalrand(randint):
    openfile = open("Results.txt", "a")
    openfile.write(str(randint) + "\n")
    openfile.close()
if __name__ == '__main__':
    from torch import manual_seed, randint
    from os import remove
    from multiprocessing import Process
    from threading import Thread
    openfile = open("Results.txt", "w")
    openfile.write("")
    openfile.close()
    memorylocalseed(manual_seed)
    memorylocalrand(randint)
    t1 = Thread(target=memorylocalseed, args=(manual_seed,), daemon=False)
    t2 = Thread(target=memorylocalrand, args=(randint,), daemon=False)
    t1.start()
    t1.join()
    t2.start()
    t2.join()
    p1 = Process(target=memorylocalseed, args=(manual_seed,), daemon=False)
    p2 = Process(target=memorylocalrand, args=(randint,), daemon=False)
    p3 = Process(target=memorylocalseed, args=(manual_seed,), daemon=False)
    p4 = Process(target=memorylocalrand, args=(randint,), daemon=False)
    p1.start()
    p3.start()
    p1.join()
    p3.join()
    p2.start()
    p4.start()
    p2.join()
    p4.join()
    openfile = open("Results.txt", "r")
    print(openfile.read())
    openfile.close()
    remove("Results.txt")

Which outputs:

<function manual_seed at 0x000001B9548751F0>
<built-in method randint of type object at 0x00007FFCA5462680>
<function manual_seed at 0x000001B9548751F0>
<built-in method randint of type object at 0x00007FFCA5462680>
<function manual_seed at 0x0000015C2685BDC0>
<function manual_seed at 0x000001BF9E54BDC0>
<built-in method randint of type object at 0x00007FFCA5462680>
<built-in method randint of type object at 0x00007FFCA5462680>

What I'm wanting is:

<function manual_seed at 0x000001B9548751F0>
<built-in method randint of type object at 0x00007FFCA5462680>
<function manual_seed at 0x000001B9548751F0>
<built-in method randint of type object at 0x00007FFCA5462680>
<function manual_seed at 0x000001B9548751F0>
<function manual_seed at 0x000001B9548751F0>
<built-in method randint of type object at 0x00007FFCA5462680>
<built-in method randint of type object at 0x00007FFCA5462680>

Without changing that it is a separate process that is "executing the module", this can be seen with the builtin function randint for context that it's possible.

This is what I get from Ubuntu (which exactly what I want also):

<function manual_seed at 0x7fae7e721040>
<built-in method randint of type object at 0x7faf397515e0>
<function manual_seed at 0x7fae7e721040>
<built-in method randint of type object at 0x7faf397515e0>
<function manual_seed at 0x7fae7e721040>
<function manual_seed at 0x7fae7e721040>
<built-in method randint of type object at 0x7faf397515e0>
<built-in method randint of type object at 0x7faf397515e0>

Any Ideas?

[Poet129]

@gabrielcarvfer

After further testing I've found linux (Ubuntu) does what I want it to automatically so I'll just use linux. However actually using those commands specifically randint bumps up the memory usage quite a bit, however this is purely just the command actually running, so no significant duplicate memory usage while using linux.

[Poet129]

In case any future readers want the code used to figure this out, this is it:

def memorylocalseed(manual_seed):
    from time import time
    openfile = open("Results.txt", "a")
    start = time()
    z = manual_seed(1)
    end = time() - start
    openfile.write(str(manual_seed) + str(end) + "\n")
    openfile.close()
def memorylocalrand(randint):
    from time import time
    openfile = open("Results.txt", "a")
    start = time()
    z = randint(9, (1, 100000000), device="cpu")
    z = z.tolist()
    z = str(z)
    z = z.replace("[[", "")
    z = z.replace("]]", "")
    z = z.replace(", " , "")
    end = time() - start
    openfile.write(str(randint) + str(end) + "\n")
    openfile.close()
if __name__ == '__main__':
    from torch import manual_seed, randint
    from os import remove
    from multiprocessing import Process
    from threading import Thread
    from time import time
    start = time()
    openfile = open("Results.txt", "w")
    openfile.write("")
    openfile.close()
    memorylocalseed(manual_seed)
    memorylocalrand(randint)
    print("Local tasks Completed.")
    t1 = Thread(target=memorylocalseed, args=(manual_seed,), daemon=False)
    t2 = Thread(target=memorylocalrand, args=(randint,), daemon=False)
    t1.start()
    t1.join()
    t2.start()
    t2.join()
    print("Threading tasks Completed.")
    p1 = Process(target=memorylocalseed, args=(manual_seed,), daemon=False)
    p2 = Process(target=memorylocalrand, args=(randint,), daemon=False)
    p3 = Process(target=memorylocalseed, args=(manual_seed,), daemon=False)
    p4 = Process(target=memorylocalrand, args=(randint,), daemon=False)
    p1.start()
    p3.start()
    p1.join()
    p3.join()
    start1 = time()
    p2.start()
    p4.start()
    p2.join()
    p4.join()
    end1 = time() - start1
    print("Processing tasks Completed.")
    openfile = open("Results.txt", "r")
    print(openfile.read())
    openfile.close()
    remove("Results.txt")
    end = time() - start
    print("It took " + str(end1) + " seconds to run both randint processes at the same time.")
    print("It took " + str(end) + " seconds to complete all processes.")

This is the linux output I got:

This is the windows output I got (I used ssh):