Questions tagged [multiprocessing]
Multiprocessing is the use of two or more central processing units (CPUs) within a single computer system
-1
votes
0answers
9 views
Python Multiprocess calls a subprocess that reads from another subprocess, still blocking with demonize set to True [duplicate]
I have a video transmitting app written in python, and I have been struggling to make the "Transmit" button, non-blocking. After the tranmission starts, the transmit button is supposed to disable, ...
0
votes
0answers
48 views
Pythonic way of generating dummy data efficiently
Trying to generate a billion rows of random data on a box with below configuration, only the data creation is running on this box. Only looking for directions on what methods can be used to improve ...
-1
votes
2answers
29 views
Creating parallel for loop in python
I looked at similar questions especially this answer. My situation is bit different, so asking question.
import os
import multiprocessing as mp
def fun1(str1):
#function def
if __name__ == '...
0
votes
1answer
15 views
Multiprocessing fails to pickle top level functions
I am trying to parallelize my task and decided to use the multiprocessing (mp) module.
I have soon learnt that mp can't handle nested functions, so I had to unnest a function similar to this:
def foo(...
9
votes
2answers
5k views
Why multiprocessing.Process behave differently on windows and linux for global object and function arguments
The following code has different output when running on windows and linux (both with python2.7)
'''import_mock.py'''
to_mock = None
'''test.py'''
import import_mock
from multiprocessing import ...
0
votes
0answers
20 views
Multiprocessing with pandas dataframe
I would like to use multiprocessing on a large pandas data frame. I want to set a column entry of that data frame based on another columns value. It is a simple labeling done with some if-statements.
...
1
vote
1answer
40 views
Multiprocessing with Python Process
I'm trying to modify a Python script to multiprocess with "Process". The problem is it's not working. In a first step, the content is retrieved sequentially (test1, test2). In a second one, it is to ...
1
vote
0answers
17 views
Python script hangs when running lstm on multiple cores
I am trying to create different lstm models on different data using multiprocessing Pool.
The program hangs when it tries to create a lstm layer.
def create_model(neurons, X):
model = Sequential()...
6
votes
2answers
1k views
How to share a cache between multiple processes?
I'm using a LRU cache to speed up some rather heavy duty processing. It works well and speeds things up considerably. However...
When I multiprocess, each process creates it's own separate cache and ...
-4
votes
0answers
20 views
Python multiprocessing equivalent for wait
Is there Python multiprocessing call that is equivalent to the wait() linux/unix system call in C?
For example, if using multiprocessing you start() a number of unrelated process and then want to ...
6
votes
2answers
3k views
Time out decorator on a multprocessing function
I have this decorator taken directly from an example I found on the net:
class TimedOutExc(Exception):
pass
def timeout(timeout):
def decorate(f):
def handler(signum, frame):
...
0
votes
1answer
18 views
Passing list to pool.map() function in Python returns each character for each item in the list
This Python 2.7 script running on Windows is not outputting as I would expect it to.
from multiprocessing import Pool
from multiprocessing.dummy import Pool as ThreadPool
def function1():
...
1
vote
1answer
22 views
Run tensorflow with one graph in multi process
I've trying to train a classifier with 5 ensemble networks. I decided to train them with different batch, so I want to create multi-processes to save my time.
Here is my algorithm design:
import ...
0
votes
0answers
24 views
Python Multiprocessing Pool Getting Stuck?
I am trying to implement multiprocessing in Python. In the below code, I have verified that calYield works properly and in a timely manner on my dataframe when run with apply, so that is not the issue....
0
votes
1answer
35 views
python multiprocessing stuck (maybe reading csv)
I am trying to learn how to use multiprocessing and I am having a problem.
I am trying to run this code:
import multiprocessing as mp
import random
import string
random.seed(123)
# Define an ...
0
votes
2answers
223 views
Using multiprocessing.Pool with exception handling
from multiprocessing import Pool
def f(arg):
if arg == 1:
raise Exception("exception")
return "hello %s" % arg
p = Pool(4)
res = p.map_async(f,(1,2,3,4))
p.close()
p.join()
res.get()
...
1
vote
3answers
63 views
Can't kill multiprocessing pool in a multithreaded C application embedding Python
OS: linux
Python version: 3.6
I'm trying to extend a C application with Python runtime. The C application uses pthread and I tried to use multiprocessing forkserver in Python runtime but faced a ...
0
votes
1answer
20 views
Analog `as_completed` in the `multiprocessing` module
I'm looking for an analog of the as_completed function (from Python 3 concurrent.futures) in the Python 2.7 module multiprocessing. My current solution:
import time
from multiprocessing import Pool
...
0
votes
0answers
19 views
Spring-batch : How to run processor with multi thread of ItemProcessor in spring batch?
Now my previous usage scenarios is like below:
using FlatFileItemReader read input stream with .txt file line by line
using ItemProcessor process per line data to invoke remote service with http
...
27
votes
2answers
27k views
python Pool with worker Processes
I am trying to use a worker Pool in python using Process objects. Each worker (a Process) does some initialization (takes a non-trivial amount of time), gets passed a series of jobs (ideally using ...
1
vote
1answer
91 views
“InterfaceError: connection already closed” when using multiprocessing.Pool on black box function that queries PostgreSQL database
I've been given a Python (2.7) function that takes 3 strings as arguments, and returns a list of dictionaries. Due to the nature of the project, I can't alter the function, which is quite complex, ...
0
votes
0answers
12 views
Telling python multiprocessing which pickle protocol should be used for serialization [duplicate]
How can I tell python multiprocessing which pickle protocol should be used for serialization?
The problem is that I want to process a large pandas DataFrame in a parallel process using ...
19
votes
3answers
10k views
Using python multiprocessing Pool in the terminal and in code modules for Django or Flask
When using multiprocessing.Pool in python with the following code, there is some bizarre behavior.
from multiprocessing import Pool
p = Pool(3)
def f(x): return x
threads = [p.apply_async(f, [i]) for ...
6
votes
1answer
3k views
pymongo MongoClient can not work in multiprocess?
I am using pymongo 3.2, I want to use it in multiporcess:
client = MongoClient(JD_SEARCH_MONGO_URI, connect=False)
db = client.jd_search
with concurrent.futures.ProcessPoolExecutor(max_workers=1) as ...
0
votes
0answers
26 views
Problems with multiprocessing.Process started from rc.local on Raspberry Pi 3
So the big picture is that I am working on a project for a museum to provide a more interactive presentation of a mural. I have a Raspberry Pi 3 Model B+ with a laser diode mounted to an Adafruit pan-...
2
votes
2answers
47 views
Is there a way to flush stdout on process termination for parallel processes
I'm running several independent programs on a single machine in parallel.
The processes (say 100) are all relatively short (<5 minutes) and their output is limited to a few hundred lines (~...
1
vote
1answer
34 views
Multi-threading on python for running 2 inter-related threads simultaneously (or may be background processing)
I want VideoCapture function from opencv to work smoothly while processing frames from it in background. As processing requires 3-4 seconds so at that time capturing video hangs.
Hence, I need to run ...
20
votes
0answers
622 views
Multiprocessing with “fork” context blocks on Linux/Intel Xeon with Python 3.6.1?
Problem description
I adjusted the code from this answer a little bit (see below). However when running this script on Linux (so command line: python script_name.py) it will print jobs running: x for ...
3
votes
1answer
42 views
Processing a list of lists in Cython, with nogil
In Python I have a list of lists as input:
input = [[0,1,2],[0,3,4,5],[0,6]]
In reality, the number of sub-lists is tens of thousands. The length of each sub-list could vary greatly, from zero or ...
0
votes
0answers
27 views
pyqtSignal OR Queue with threading Timer for heartbeat?
I'm experimenting with creating a HeartBeat(threading._Timer) class that is to notify the main class if a program is still alive or not.
I was originally trying to leverage the pyqtSignal from PyQt4....
0
votes
1answer
9 views
Does lock.acquire blocks processes in non critical section
Maybe I didn't read the docs that well, but I didn't find more information about how lock's or rlock's aquire work ... does it block all processes no matter what statements those processes are on (...
0
votes
3answers
70 views
Multiprocessing Running Slower than a Single Process
I'm attempting to use multiprocessing to run many simulations across multiple processes; however, the code I have written only uses 1 of the processes as far as I can tell.
Updated
I've gotten all ...
3
votes
1answer
37 views
python multiprocessing with different functions
I can't find an appropriate method for the next task:
I have an input massive [x1, x2, x3 ... xn] and function that I want to apply to every x from massive f(xi, param) so there are eight possible ...
1
vote
1answer
43 views
Can a single Android phone connect to multiple bluetooth devices?
I am about to create an "Intelligent" motorcycle helmet which will be able to play music, discuss with the passenger, send GPS information, notify of police, speech recognition etc... I have a few ...
android
0
votes
1answer
44 views
Python multiprocessing issue, complexity does not increase execution time [on hold]
I am fiddling with Python's multiprocessing capabilities along with other features such as maps and lambdas. Python version is 2.7.8.
The code in question follows. A worker function that iterates ...
1
vote
2answers
2k views
Flask application not working when using thread pool (using UWSGI)
Im implementing multithreaded functionality in Flask using multiprocessing ThreadPools.
My problem is that is does not work. When the threadpool is activated the application halts.
If I restart ...
0
votes
0answers
8 views
pgmpy - multiprocessing of add_edges_from
pgmpy is a library for working with Bayesian Networks and Markov Models.
As part of this toolbox, add_edges_from adds bulk edges in a given model. Since I am processing large data sets, I am thinking ...
0
votes
0answers
17 views
Correct way to visualize Unix job?
Is this the correct way to visualize a Unix job? And is there anything I'm leaving out out? I'm working off the shell definition - e.g. that a job is a process group. I'm writing a post on parallel ...
1
vote
1answer
32 views
python multiprocessing pool not work when we run many tasks
Example: I have a CPU with 2 threads so I have 2 workers with this code.
tasks = ['1.txt', '2.txt', '3.txt', '4.txt', '5.txt']
pool = multiprocessing.Pool()
pool.map(myfunc, tasks, chunksize=1)
pool....
2
votes
1answer
37 views
What is the most efficient way to apply multiprocessing to unique categories of entries in a pandas dataframe?
I have a large dataset (tsv) that looks something like this:
category lat lon
apple 34.578967 120.232453
apple 34.234646 120.535667
pear 32.564566 120....
-2
votes
1answer
38 views
Python multiprocessing slowed down even in mono-thread
I want to run the same code in parallel using multiprocess.
My process code run in 8 minutes alone. In 10 minutes when using the "force mono-thread" thing. But when I run 24 of them in parallel, ...
0
votes
0answers
10 views
Reporting on child process errors in pathos
Using pathos to pickle complex and heavy py 3 objects.
Using pipe to fork these processes I do not need to return results as these are serialized independently. What I need is try except framework ...
0
votes
0answers
18 views
MCMC sampling using emcee is running extremely slow on intel xeon phi processor
So, I have the following code for MCMC for the parameters of a Gaussian process
code:
walkers = 250
pos_min = np.concatenate((np.array([self.initial_scale, self.initial_noise]), np.zeros(...
2
votes
2answers
44 views
python executor spawn tasks from done callback (recursively submit tasks)
I'm trying to submit further tasks from result of a task that was done:
with concurrent.futures.ThreadPoolExecutor() as executor:
future = executor.submit(my_task)
def callback(future):
...
0
votes
0answers
23 views
EOFError: EOF when reading a line python3 multiprocessing queue
I've been looking online for some hours now to solve the following problem but with no luck at all. All situations are a little different than mine.
My code:
from multiprocessing import Process, ...
3
votes
1answer
53 views
Getting some form of keras multi-processing/threading to work on Windows
In order to speed up data augmentation for training a neural network, I am trying to have some form of parallel processing for feeding my GPU with data. At the moment the limitation is how fast I ...
1
vote
1answer
23 views
dask.delayed results in no speedup
I am trying to get into Dask. For that I attempted to parallelize some time consuming sequential code I got. The original code is this:
def sequential():
sims = []
chunksize = len(tokens)//4
...
3
votes
0answers
49 views
How to throw an exception for a long running line of code inside a thread?
I have a program that splits the processing of a large dataset into parallel processes with billiard like this:
import billiard
from billiard import Queue
def process_cursor(skip_n, ...
0
votes
0answers
17 views
Correct way to implement multiple pipes in Python
I am computing the overhead time for creating processes (multiprocessing module) and sending data with pipes.
def worker_process_pipe(pipe, image):
pipe.send(image)
pipe.close()
def ...
0
votes
0answers
13 views
Pathos queuing tasks then getting results
I have a computing task which effectively is running the same piece of code against a large number of datasets. I'm wanting to utilise large multicore systems(for instance 72 cores).
I'm testing it ...
