Transfer Learning for Image Aesthetic Assessment

The project aims to develop transfer learning methods for deep convolutional networks. In this framework, besides the main database, which allows supervised learning, we will consider a second database, larger in volume, that contains examples either without annotations or with labels, but for another, related, problem. The transfer learning involves extracting useful information from the second database to increase performance on the primary set, in a simultaneous process.
For the development of the transfer learning algorithm, the characteristics of the convolutional networks to form descriptors on the intermediate layers, respectively to be sequentially trained are exploited. The central algorithmic principle is the partitioning by clustering (if the data does not have labels) or by classifying the secondary data set, where the descriptors of data are extracted on an intermediate layer. The network is required to create descriptors that provide wide margins between the data of one group and the centroid of another group. The groups can be associated with the classes from the primary database and the method of calculating the wide margin will be explored. Secondly, for the harmonization of the two databases we will use regularization by injecting a random perturbation, in an annealed process, into the global gradient.
From an application point of view, we aim to estimate the aesthetic value of a photograph. The aesthetic value aggregates, consciously or unconsciously, how pleasing an image looks. For high performance, in this direction we will exploit the previously developed transfer learning algorithm. The application is useful in commercial area to offer customers visual quality.

Objectives and achievements

The project aimed to work on two directions:

• [O1]. Transfer Learning and CNN layer structuring
• [O2]. Image Aesthetics which is to estimate automatically the aesthetic value of an image.
The directions were aimed to be distinct conceptually, but also to communicate between them.

[O1]. On the transfer learning direction - the objective was to develop the individual concepts and combine them in robust algorithms that are able to efficiently transfer information from a secondary database to a primary subject one. The individual concepts identified at the proposal stage were:

• [O1.1] Large margin . This means to ask that the data from the large unlabeled set enforces large margins between pseudo-classes using on the encodings (taken from some layer before the decisional one) to cluster according to classes and the distance between a point and other cluster centroids should be used.
  

  Achievements

  • - We have showed that this idea is feasible and useful for more difficult (from a machine learning point of view) problems such as face expression. The resulting algorithm was named Margin-Mix and published in ref [1] [here]. A variation, also on on two databases, this time even with distinct annotations on the two databases is proposed for publication in ref [R4] [here]
  • - We have showed that this idea is also useful for training in supervised manner a single database which is more difficult. The algorithm was named Soft-Mix and proposed for publication in [R3] [here]
• [O1.2] Annealed regularization which is to injecting random perturbation into the overall gradient in a two database problem, one is able to cope with a bias between the two databases.
  

  Achievements

  • - We have showed the feasibility of this idea in various experiments in either aesthetic evaluation [4] [here] or in [6] [here] in a single database scenario.
  • - The idea of injecting randomization in two database was shown to work for other more difficult problems such as face expression is proposed for publication in [R2] [here]
• [O1.3] Semi-supervised learning via clustering loss this is a derivation of objective [O1.1] with direct application in semisupervised learning
  

  Achievements

  • - The concept is strong enough to boost significantly the performance of small networks as showed in [7] which is [here]

[O2]. Image Aesthetics. The objective here, was given an image to provides scores from aesthetic point of view. Two approaches are taken commonly: one is to give a holistic and unique grade to the overall image; the other is to asses specifically various aesthetic elements.

Achievements

Both approaches have been attempted. In a structured way achievements are:

• - Performance on the holistic grade are in [4] [here] and respectively [R3] [here]
• - Performance on the attributes are in [5] [here]
• The available databases did not unify the two approaches. To work on that, we have used many observers to annotate a set of images with attribute. Those images had already holistic annotation. The process of anotation finished in June 2022 and results are still under analysis. The results will be made public as A-AVA database in the Download section.