Project Overview

The Google Summer of Code has ended.

I have made a final report with everything I have done this summer: http://docs.google.com/Doc?id=dgmqj5ct_636556grfv

I hope these tools will prove useful :)

[analyzing fragmentation] Looking good

Hello,

I have finished tweaking my plotting tool and today I have played with it, ran some programs, analyzed some graphs.

I am going to show you the steps in which I can find "hot spots" (which increase fragmentation) in a program's code with the help of the graphs:

First, here is an example plot for nautilus:


Now I am going to detail a plot of gnome-panel:


What I'm looking for are places where the fragmentation level raises quickly and then find the allocations responsible for that.

Here is one example: I am zooming in to the region where the fragmentation raises very quick and this is what I find:


Now, I look at the X axis and see when the allocations took place and this way I identify them:

- g_main_context_iterate (gmain.c:2696) : 78.85% fragmantation
- g_main_context_dispatch (gmain.c:2615) : 78.89% fragmentation
- g_main_dispatch (gmain.c:2063) : 78.77% fragmentation

Let's look at another region:


Here, I found that signals are the ones to blame:

- g_signal_emit (gsignal.c:2243)
- signal_emit_unlocked_R (gsignal.c:2478)
- g_signal_emit_valist (gsignal.c:2209)

While running different programs, I have noticed that indeed signal handling is the main reason for fragmentation. I am going to look into that, but right now I know nothing about signal handling in GNOME. I would appreciate some documentation and some hints on where to start, to understand the big picture.

I still need to find a way to download the exact sources of the installed packages, so that I can look at the correct lines in the source code given by valgrind. ( I am running Ubuntu, btw )

One more thing: I have been thinking ... I am measuring the fragmentation level at the point when an allocation gets freed, but maybe it is more relevant if I measure the fragmentation level immediately after the allocation is created ?

I'd like to hear some feedback from you :)

Some interesting results

I have some nice results to show you. I have succeeded in modifying Valgrind's massif and obtained some nice results, which I have plotted using gnuplot.

Below is a graph, where X represents time of allocation (measured in instructions) and Y represents the lifetime of the allocation. All allocations while a program is executed are plotted. In the examples below, I have used gedit.



The idea is to replace short consecutive allocations with static buffers.

The image I have posted is of poor quality, I don't know how to save the graph in the format from inside gnuplot (the internal viewer from gnuplot permits for a high zoom-in, for a detailed view), but I will anyway use gnuplot's viewer.

I pushed the script I used to make the graphs to my git repository, but you will need to rebuild valgrind with my modified ms_main.c in order for the script to work.

My first GUADEC

I got home on Wednesday (16th) from Turkey and I'd like to express some opinions about my first GUADEC.

It was amazing !! I could have never imagined GNOME is such a wonderful community. You people are great!

I met a lot of people in Istanbul, some really great hackers and others students, like me - well actually a little bigger :P and I had a really good time with Simos, Clemens, Felix, Cosimo, Diego and many others. I also had some interesting conversations with great hackers such as Michael Meeks, Brian Nitz, Hans Peter and, of course, Federico, the person to which I look up the most :)

Finally, I wanted to thank Baris for a great organization and some really nice parties - very good job !

Turkey was great, unfortunately I didn't have time to visit Istanbul too much, but at least I managed to see Hagia Sofia and the Blue Mosque - amazing... After GUADEC I went to the Mediterranean Sea and swam in one of the clearest waters in Europe.

I almost forgot about the food - I put on 3 kg while in Turkey, so you can imagine how much I liked it :D (especially the Turkish baklava)

I can't wait to see you people again at the next GUADEC, or perhaps GUADLAC ? :)

My work so far

So far, I have been working on the tools I mentioned earlier, to create images of memory fragmentation.

To do this, I have hooked glib's g_malloc, g_realloc and g_free (in mp.c, based on Stefan Kost's work) and logged every call to these functions to a file, containing the address and size of the newly allocated memory blocks.
For g_free() I could not determine the size of the freed memory, so I have created free_size.c, which basically adds all malloc's to a hash table and when free is encountered, I just search the hash table for its size; as for realloc, it is just like free+malloc.

To use these hooks with a program, I make use of LD_PRELOAD, which points to a shared object built from mp.c

Finally, to generate the images, I used Stuart Parmenter's python script. Here are 2 image samples:

(grayscale = fragmented; cyan = unused memory)

I am not proud of how they look, but these are just initial drafts. The important thing is that they prove my tools work.

The next step in my project is to make similar hooks for the rest of the allocators and make customized images for them. I will need to make some changes to the python script, because I am not completely satisfied with how the images look right now :)

I will keep you updated.

Overall Description

I thought I'd start with an overall description of the project and then go into details.
So here's the plan:

At first I am building some tools that generate images showing the memory fragmentation caused by the allocators at specific times in the program execution. I will try to use different colors for different allocators (malloc, g_malloc, realloc, g_realloc, gslice), so as to help me determine the main causes of fragmentation and then act accordingly.

You will find in the References section links to the project description, my mentor's blog and the repository where I host my code.

My first post

This is my first blog, better yet, my first personal web site :)

I wasn't really into blogging, I never really had a good reason, but now, an opportunity has risen: I have created this blog in order to post my progress on my GSoC project and intend to use it after GSoC finishes, to track my open source contribution.

I am currently working for GNOME and intend to continue my work for them in the future - so far they have been nothing but nice to me, especially my mentor, Federico Mena Quintero.

In the posts to come, I will write about my progress so far with my current project.