Newest 'parallel-processing' Questions - Stack Overflow

Questions tagged [parallel-processing]

Parallel processing is, in sharp contrast to just a Concurrent processing, guaranteed to start / perform / finish all thread-level and/or instruction-level tasks executed in a parallel fashion and provides a guaranteed finish of the simultaneously executed code-paths.

1
vote
1answer
19 views

MsTest parallelization

Im running two tests that share resources in parrallel. I expected that the Test1 would fail, however it turned out that MsTestV2 creates a new instance of the test class and therefore the member ...
0
votes
0answers
9 views

os.system() calls to mpirun don't work after calling a script with python

I am trying to run mpirun on a python code that has already been run via console, a jupyter notebook or from Pycharm's console. All the code run goes fine until the call to the mpirun process, which ...
0
votes
1answer
18 views

Using multiprocessing module to runs parallel processes where one is fed (dependent) by the other for Viterbi Algorithm

I have recently played around with Python's multiprocessing module to speed up the forward-backward algorithm for Hidden Markov Models as forward filtering and backward filtering can run independently....
1
vote
2answers
36 views

How to run bash functions in parallel and merge all output to one string in order?

I'm writing several bash-prompts functions in the style of oh-my-git and I want them to run in parallel and merge their output in order. Speed is of the essence and I want to avoid writing to any file ...
1
vote
2answers
42 views

control conditional Openmp in #define macro

I want to use a #define flag to control using openmp or not. Since #pragma can't not be inside a #define, so I tried #define USE_OPENMP // Toggle this on/off #ifdef USE_OPENMP #define OMP_FOR(...
0
votes
0answers
10 views

Output from ASyncResult

I'm finding a very weird problem regarding ipyparallel library: using the function apply I'm obtaining as output and object called ASyncResult followed by a list which is the correct output that I'd ...
0
votes
0answers
8 views

Using multiprocessing Queue to prevent corrupt plots from being generated

With matplotlib backend set to 'PDF', I have incorporated plotting into a function. The central theme of the function is to produce plots, like so: def func(arg1,arg2,arg3,num): try: fig =...
0
votes
1answer
20 views

A fast solution to obtain the best ARIMA model in R (function `auto.arima`)

I have a data series composed by 2775 elements: mean(series) [1] 21.24862 length(series) [1] 2775 max(series) [1] 81.22 min(series) [1] 9.192 I would like to obtain the best ARIMA model by using ...
0
votes
0answers
21 views

Python parallel processing across all cores on multiple nodes?

We have some python code that runs a scorecard. Due to the size, we are trying to utilize 128 cores across 3 nodes in Windows. However, currently the code only utilizes one node. Is there a simple ...
0
votes
0answers
20 views

How to tell which node is executing code as it executes in Slurm?

I am very new to Slurm and distributed/parallel computing so hoping someone could shed some light on my issue but bearing in mind this might be a bonehead simple problem to solve. I've set up a ...
1
vote
2answers
22 views

How to process multiple files through a program in bash using a predefined number of processes?

I'm running a computationally heavy program on a list of files in bash. If I do them one at a time, I do not utilize my computer power, yet if I append the & symbol to the command to run them in ...
0
votes
0answers
17 views

I want to execute multiprocessing to align images in directory but showing no output

I am aligning different images using the base(reference) image, but it takes lots of time if I provide directory. I was planning to multi-process the task. The task shows no error but its not giving ...
0
votes
0answers
14 views

Multiprocessing nested loop in python

I am very new to complicated coding in python but I want to improve my code to perform much better and much faster. Searching for that, I came across 'multiprocessing', which I tried to implement, but ...
0
votes
1answer
22 views

ARIMA forecasting in R is taking too long. Please suggest vectorized R function for ARIMA, or parallelization or something else [on hold]

The ARIMA model that I am running is taking a lot of time. I am using the auto.arima command in R for the forecasting. Please suggest the vectorized version (just like log() is a vectorized function ...
3
votes
1answer
36 views

How To Run MPI Python Script across multiple nodes on Slurm cluster? Error: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1

I'm running a script on a Slurm cluster that could benefit from parallel processing, so I'm trying to implement MPI. However, it doesn't seem to allow me to run processes on multiple nodes. I don't ...
0
votes
1answer
19 views

Multiple threads reading unique records from the same table with Entity Framework

We have multiple concurrent threads that simultaneously "claim" unprocessed records from a database table to work on. In order to ensure no threads end up claiming duplicate records, we use a query ...
0
votes
0answers
41 views

Is executor service, concurrent or parallel? [duplicate]

There are two separate threads in my code that are using there own Executor's SingleThreadExecutor. I wanted that both those services should run parallelly, but what I observed that one service haults ...
0
votes
0answers
6 views

How to process WARC files faster?

I'm extracting links from WARC files with Python on single server and using multiple cores with running multiple instances of parser script. What I want to know is, is it possible to process/parse ...
1
vote
0answers
17 views

Parallel processes overwriting progress bars (tqdm)

I am writing a a script in Python 3.7 that launches several parallel tasks using multiprocessing.Process (a task per core). To track down the progress for each process, I used the library tqdm which ...
0
votes
0answers
6 views

How to process mongodb documents with N applications instances in parallel without duplications?

I'm planning an application that can run in multiple instances with a goal of processing documents from mongodb. For example each instance of the application would read 1000 documents and then process ...
1
vote
0answers
35 views

Assigning threads equitably to columns in parallel processing

I'm currently working in a program that applies Gaussian Blur to an image, I assign a thread to a certain number of columns of my image in order to process it using a parallel model. The problem I'm ...
0
votes
1answer
31 views

How to get one correct value from a function which is parallelized by multi-threads

For faster calculation, trying to make my method parallelized by 4 threads. Where threads are making 4 separate calculation whether I was expecting concurrent operation and on single variable. #...
-3
votes
0answers
28 views

How can I process 1 TB file with Flink?

I have a 1 TB data file and a Flink cluster with 8 GB of RAM. How can I sort the data in this file using Flink? What other alternatives can I use for this?
0
votes
2answers
21 views

Applying the pool.starmap() function with multiple arguments on a dict which are both passed as arguments inside the .starmap()

I'm currently trying to apply parallel processing on a script. The following is just to simulate the current script I am working on. Basically, I want to call the Pool.starmap() function on a function ...
1
vote
2answers
39 views

Acheive concurrency in Kafka consumers

We are working on parallelising our Kafka consumer to process more number of records to handle the Peak load. One way, we are already doing is through spinning up as many consumers as many partitions ...
-1
votes
1answer
33 views

How to call some methods parallelized and get information from this methods?

I am developing an API, in this API I have a service with a method and this method has to call three methods in different repositories. Each of this methods have a Https request, the call has a ...
1
vote
1answer
22 views

Why is there no output in parallel assignment of h5py dataset?

I am trying to generate an hdf5 dataset from code that runs in parallel, but when I read the dataset generated, it is blank, all entries are zero. I have replaced the parallel code with a sequential ...
0
votes
1answer
26 views

How to pass down parameters to a multiprocessing loop

I want to do a Hartree-Fock kind of computation. In short, it's a self-consistent matrix convergence problem. I start from a matrix H(0), from which I construct a Fock(0) term. Now I define H(1) = H(0)...
0
votes
0answers
20 views

Async annotation and parallel processing

Here is the environment first: A service publishing high amount of messages to a queue. This is a Spring boot application running in a container and I have access to it (means it is possible to ...
2
votes
2answers
16 views

How can I import thread package in Python 3?

I want to import thread package in Python 3.6. But this error is occurred: import thread ModuleNotFoundError: No module named 'thread'
0
votes
0answers
62 views

Async call with awaited Semaphore create Deadlock

From last few days, I am facing an intermittent deadlock from the below code snippet, It occurs only when we use async call, my Api call hangs for infinite period of time, the worst thing is I am not ...
0
votes
1answer
27 views

Encode sentence as sequence model with Spark

I am doing text classification and I use pyspark.ml.feature.Tokenizer to tokenize the text. However CountVectorizer transforms the tokenized list of words to bag of words model, not the sequence model....
1
vote
0answers
35 views

Why pmap(fib,array) results in type instability when map(fib,array) is type-stable?

I am playing with pmap, but noticing that it transforms my type-stable Fibonacci function (recursive version) in type unstable.. wondering why and if there is something wrong I am doing: julia> ...
1
vote
0answers
29 views

Processing and creation of array in numba.cuda device function

I want to pass to device function a slice of an array and then create there some new array and return their combination. But it seems not the general way to solve such problems with cuda because numba ...
0
votes
1answer
23 views

How to parallelly access files from shared location by 2 instances of a Springboot Service?

There are 2 instances of a spring boot service which read files coming to particular location. If a new file is added in the location then both the services read the file together. The aim is that the ...
2
votes
0answers
32 views

Faster NHibernate loading in multi-tenant environment

I need to access hundreds of SQL servers with the same set of schemas using NHibernate. Each customer has several different databases for different functions. Each customer stores their own connection ...
0
votes
1answer
19 views

In python, run in parallel an executable .exe without waiting the .exe to be finished

This is just an example, but more situations might arise. Suppose you have an executable from a "myscript.py" file, with a kind of infinite while-true, but it is actually a condition extracted from ...
0
votes
0answers
13 views

sklearn with parallel backend sends logs/output to console instead of notebook cell output

I'm using GridSearchCV to do some analysis for a model (same problem happens with learning_curve btw) and have verbose parameter set to 2. The only output I get in the notebook is some minor progress ...
0
votes
0answers
13 views

Error in makeMPIcluster :a cluster already exists 1

I'm trying to parallel my code. the first part of my cod was paralleled by Rmpi and work correctly. now I want to parallel the second part(Estimate generalization error) by parallel and Snow package ...
1
vote
0answers
93 views

Global memory access coalescing in CUDA - Maxwell architecture

I have code for matrices multiplication running on my Geforce 940m (Maxwell architecture) with CUDA compute capability 5.0. I have used NVIDIA Visual Profiler to measure number of global load ...
0
votes
1answer
44 views

Multi-Threading in Eigen (OpenMP is not used)

I have a problem in using multi-threading in Eigen library. This is my code: #include <QCoreApplication> #include <iostream> #include "Eigen/Core" #include <QDebug> using namespace ...
1
vote
1answer
34 views

how do I get MPI profiling turned on with the MPE library?

I am working on a free software application targeted at meteorologists and climate scientists called PIO: https://github.com/NCAR/ParallelIO. As implied by the name it is a parallel I/O library using ...
1
vote
0answers
21 views

Reasons why serial algorithm is slower than parallel version without extra threads? [duplicate]

I made a serial MergeSort and a parallel MergeSort. Both divide the input list in a left and right half and call sort on those. The source code is almost identical; the difference between them is that ...
1
vote
1answer
31 views

Distributin Ruby Code on Multiple Threads with Batches

This is a kinda stupid question but, but I'm having some issues distributing the correct amount of data that a worker needs to process. For a bit of context, I have 60 Resque Processes (workers), ...
0
votes
0answers
24 views

Reference iterator in parallel loop

I'm trying to reference the iteration value i in a parallel loop using Joblib. I'm getting the error NameError: name 'i' is not defined. Below is my code: def sub5(): curveplots = [] nodes = ...
1
vote
1answer
37 views

Julia - Is it possible to use a progress bar and Threads.@threads in same for loop?

Suppose I have this for loop: for i in 1:100000 1+1 # Let's do some long computations here end Those two for loops work: Threads.@threads for i in 1:100000 1+1 # Let's do some long ...
0
votes
0answers
29 views

Pros & con of parallelizing task at higher vs lower level

I have a Cherrypy server. Some of that server's job is to crunch data (let's call it BigCalculation) return the result to the client. The raw data is a numpy array (3 dims) and could ultimately be ...
1
vote
1answer
33 views

How to distribute a loop to run one iteration per core

I am running a semi-large loop in a shell script on a server with 30 cores. It looks like the one below. I would like to distribute this loop to run one iteration per core: python sim_fine.py model1 ...
0
votes
0answers
9 views

How to execute pytest paralelly through xdist plugin?

How can I run my pytests parallely on multiple custom environments ? I have pytest-xdist as well. Not sure if this plugin helps test_test.py @pytest.mark.env("env1", "env2", "env3") def test_check(...
1
vote
1answer
14 views

Can't parallelize creation of a Materiazed View

I',m tryng to create a Materialized View referencing a remote table with over 8 million records. I put the clausole "PARALLEL 8" in DDL MVIEW statement and the hint "/*+ PARALLEL(8) */ in the MVIEW ...