[Israel.pm] patterns findings

Evgeny evgeny.zislis at gmail.com
Wed Mar 18 10:20:52 PDT 2009


Another approach might be to write a log analyzer using some kind of
parser generator, it can create statistics regarding which patterns
are the most common and are canditates for optimization.

It might take a couple of days to write, but something simple can be
done with Parser::RecDescent or Parser::Yapp. And it can also be used
for more in-depth log analysis when enhanced.

- evgeny


On Wed, Mar 18, 2009 at 5:17 PM, Avishalom Shalit <avishalom at gmail.com> wrote:
> if you want a generalized pattern matcher (simple generalizations may
> include white spaces, complex ones may include "number" patterns or
> "Last Name" patterns)
> i do not think a module is there just for you,
> what i would recommend is to start going over it manually, creating
> your own list of patterns.
> (in which case enumerate them, and count re occurrences in an array. )
> and once you add something to the list, grep for the lines that don't
> match any of your patterns to see what you have left.
>
> i would guess that pretty quickly you would cover over 90% of your logs.
> then it would get slower, perhaps never reaching 100%.
>
> in one word -
> manually
>
>
>
>
>
>
> 2009/3/18 Gabor Szabo <szabgab at gmail.com>:
>> 2009/3/18 Yossi Itzkovich <Yossi.Itzkovich at ecitele.com>:
>>> Gabor,
>>> The problem is that I don't know the patterns - I want the script to find.
>>> Let me explain the need:
>>> We have a big tracing log of SQL queries to DB. We want to analyze it and find if there are repeating sequences of same  queries, and optimize them (make one big query, or change application code).
>>
>>
>> Well, if you give a real example instead of an abstract one then you
>> can get more help.
>>
>> I guess there are tools out there to analyze such log if that is a standard log.
>>
>> So you could either show us a few lines of the real log file or tell
>> us what tool
>> produced it. Is it the logging message of DBI ?
>>
>> If you really have no clue of what a repeating string can look like
>> then just go with
>> if ($str =~ /(.+).*\1/) {
>>    print $1;
>> }
>>
>> or better yet tell the thing you want the repeating string to be at
>> least 10 characters long:
>>
>> if ($str =~ /(.{10,}).*\1/) {
>>    print $1;
>> }
>>
>> Gabor
>> _______________________________________________
>> Perl mailing list
>> Perl at perl.org.il
>> http://mail.perl.org.il/mailman/listinfo/perl
>>
>
>
>
> --
> -- vish
> _______________________________________________
> Perl mailing list
> Perl at perl.org.il
> http://mail.perl.org.il/mailman/listinfo/perl


More information about the Perl mailing list