Researchers from the University of California at Santa Barbara (UCSB), the University of Rome 3 (Roma Tre), and the University La Sapieneza (Rome) have teamed up to solve the problem. They are presenting results in a paper, "Texture Based Classification and Segmentation Using the Laguerre-Gauss Transform," at the 2nd International Symposium on Image and Signal Processing and Analysis to be held at Pula, Croatia, June 19 to 21.

On September 26, 1997, an earthquake registering 6.0 on the Richter scale and coming nine hours after an earlier tremor, shook the 13th-century Basilica of St. Francis of Assisi in northern Italy.

Four people were killed, two Franciscan friars and two technicians inspecting and photographing the Basilica's ceiling in the wake of the initial tremor. The vaulted ceiling fell on them. And the Cimabue fresco of St. Matthew and the Giotto fresco of St. Jerome crashed into tens of thousands of pieces.

Think of the problem as computer assembly of a jigsaw puzzle whose pieces have been thoroughly chewed by dogs so that the shape of pieces cannot be relied upon to match pieces. The fresco fragments cannot be fitted together by shape. What's left, then, for guidance, are mainly color, but also, detail.

The frescoes were painted about 750 years ago on basically a cement surface. Some cement pulverized when the frescos shattered. That dust settled on the fragments and had to be washed off. But washing the fresco fragments restored them to a color different from the color in photographs taken just before the ceiling collapsed, because the intact frescoes had accumulated a dulling film from sources such as candle soot. One of the image-processing problems is to look at the color of the washed fragments and correct or "filter" that color so that it is akin to the color in the photographs.

Color filtering and color matching are specialties of Sanjit Mitra's research group. Mitra is a professor of electrical and computer engineering at UCSB. That expertise drew the Italian professors to consult with Mitra, but his contribution to the research problem -- the focus of the paper being delivered in Croatia -- has to do with attacking the problem by using textures, instead of individual pixels or picture elements, when searching a huge database of fresco fragments.

Before the work with Mitra, the Italian electrical engineering professors, Giovanni Jacovitti, Alessandro Neri and Marco Carli, analyzed the research problem of reassembling the fragments via computer. The photographs of the intact ceiling would act as a template much the way the picture does on the box containing jigsaw puzzle pieces. The first thing Jacovitti, Neri and Carli decided to do was conduct a feasibility study.

They had an artist paint a replica of the Cimabue, which was photographed in such a way that the resultant photographs would be like those of the original ceiling. The conditions that had to be simulated included lighting, distance from floor to the Basilica ceiling, and the curvilinear shape of the vaulted ceiling.

Then they dropped the Cimabue copy so that it would smash into pieces that could be photographed and scanned into the computer. In the meantime they wrote an algorithm and a program to instruct the computer how to search the database of pieces and identify by matching color and detail the location of pieces on the photographic template.

According to Neri, "Our method works by having the computer reposition the pieces with the highest confidence level. Then successive searches work down the levels of confidence."

The experiment showed that the computer could reposition between 90 and 98 percent of the fragments.

The next step, Neri explains, was to attack the problem of the quality of the actual photographs of the Basilica ceiling. Analyzing that problem provoked the Italian engineers to a key realization having little directly to do with the shattered frescoes.

They realized that in order to restore damaged artworks digitally by comparison with photographs, the photographs have to be taken according to precise guidelines. And the conditions under which the photographs are taken have to be noted along with each photograph -- i.e., time of day, lighting, camera position. One can easily imagine storing that information digitally along with each photograph.

Mitra's graduate student Serkan Hatipoglu has been studying the problem of image retrieval from a vast database of photographic images for four years. "I have been trying to find features to distinguish images, so that I don't have to look at all the image information," explained Hatipoglu. "I just extract features from texture patterns and compare those features to find similar images in the database."

Hatipoglu's efforts long preceded his involvement with the Assisi frescoes. Why was he working on the problem? Mitra and his student have been envisioning Internet search engines of the future. The idea is to search by image -- i.e., said Hatipoglu, "Give me a barking dog or a rotating yellow bowl, but search not by using words but image features."

Carli worked as a visitor in Mitra's UCSB lab last summer and returned with Neri in February. Mitra suggested sorting by texture-based classification and segmentation using the Laguerre-Gauss Transform algorithms and program the Italians had written.

Said Mitra, "When Marco explained last summer that they were doing a pixel-based analysis, I suggested they also use texture. In other words, not just one pixel but blocks of pixels. Blocks afford a much more robust approach, so that the process of searching and sorting is faster."

Hatipoglu has implemented that approach on searches for cement.

The pieces of a jigsaw puzzle are three-dimensional; there is the large two-dimensional surface area of the picture, and the slender third dimension of depth, i.e., the thickness of the pieces. All the pieces have the same thickness, and even though the pieces have curved as well as straight sides, all the sides are themselves straight up and down. The sides are at right angles to the surface.

The breaks between the fresco fragments are not straight up and down, but beveled. So a fragment in hand can show a little color of the original painting and a lot of cement along the plane of the break. Hatipoglu can search the fragments, pick out the cement, and subtract it, leaving paint for comparison of both color and detail. The Croatia paper reports this success.

So far, the computer-aided restoration techniques have made little contact with the process of the actual restoration of the Cimabue and Giotto frescoes. Neri and Carli speculate that the idea of a computer doing the looking may have seemed initially perhaps unsightly to such visual professionals. The electrical engineers emphasize that what they are proposing is an aid to, not a substitute for, the human restorer.

Their work on the Assisi frescoes has convinced Neri and Carli that not only Italy, but other earthquake-prone places should conduct a systematic photographic survey of artworks in such a manner that image-processing techniques can readily be employed via computers to reconstruct damaged works. They point out that such a survey would also enable accurate tracking of deterioration caused by pollution.

In the meantime, the UCSB-Rome 3 collaborators are casting about for other applications for their findings, and their eyes are on the Internet.

All of the above participants in the project are authors on the Croatia paper. - By Jackie Savani

[Contact: Sanjit Mitra, Alessandro Neri, Marco Carli, Jackie Savani]

21-Jun-2001