[Israel.pm] performance problems - Win32::Process which runs perl

guy keren choo at actcom.co.il
Fri Sep 15 10:59:35 PDT 2006


On Fri, 2006-09-15 at 18:01 +0300, Sagiv Barhoom wrote:
> On Friday 15/9/2006 00:26, guy keren wrote:
> > On Thu, 2006-09-14 at 00:08 +0300, Sagiv Barhoom wrote:
> > did you actually measure this (not on _your_ program. on a minimal
> > 'hello world' perl program), or just guessed?
> No, but AFAIK Linux Process menagment is much better.
> but you gave me aa idea, tou remind me that when I had such a problem on Linux 
> machine I added a sleep(1) at the child process and that lowered the CPU by 
> more then 30% .

when doing performance optimization, "i assume" is a word that should be
immediately followed by "i tested" and then "i measured" and "here are
the results". otherwise, you're only doing hand-waving (or mail-waving).

> for heuman user adding 1 sec at the begininig of the runtime is OK in that 
> case - so I think this will solve the problem.
> use SOMe::PM ();
> helped to get it to 95%-98% CPU usage.

i've no idea what 'SOMe::PM();' is. i hope you do.
i also didn't understand what you said in the last sentence (starting
with 'for heuman' - even if you meant 'human' it is not inteliggable,
since you are not writing a user-interactive program here - or are
you?).

>  I will try that next week.
>  
> > changing your design. don't generate a script per record in the database
> > table. since i don't know what your code does, how can i suggest
> > alternative designs?
> >
> Yet I will be happy to read alternatives.
> the progrem  work like this:
> Start:
> 	Get records for DB
> 	foreach $rec (@records){
> 		create from the record  a script;
> 		wait until less then $MAX_NUMBER_AT_SAME_TIME chidren scripts are running
> 			(at the same time)
> 		run that script you have just created;
> 	} 
> End;

this doesn't tell me much. you already told us what you were doing at
the level of 'read record, generate script, run script'. what i am
asking is what does this script do? why do you have to generate a script
and run it dynamically? only because it looks lovely? if something
doesn't work fast enough, you should consider throwing the entire design
away and doing something different. for example, if your generated
scripts tend to do one of a prefixed list of operations, just prepare
them a functions in advance, and invoke functions instead of running
processes.

if they generate SQL commands - consider using dynamic SQL (thought
dynamic SQL is much much slower then static SQL, so whether this will
help or not is a question for a good DBA).

thus, if you want more help, tell us in more details what your table is,
how the records look, and what do the generated scripts do.

> 
> > if you are trying to execute 1000 programs you can:
> >
> > 1. execute them 1 by 1.
> > 2. execute them all at once.
> > 3. anything in between.
> 
> as you can see I am doing No. 3
> 
> > in general, when running CPU-bound programs on a machine with N CPUs,
> > executing more then N programs at once is stupid.
> 
> yes - but since I got IO here - I belive Parallelizm is te smart choise in 
> that case.

did you _test_ this, or do you remain a pure believer? you should test
this in various configurations. play with the $MAX_NUMBER_AT_SAME_TIME
variable - chekc with the value '1', then '2', then grow and see when
your performance peaks.

--guy




More information about the Perl mailing list