K-L Divergence,entropy and mutual information of experiments in the intuitionistic fuzzy case

Abstract

In the paper, we define the Shannon-typeentropyof experiments in the intuitionistic fuzzy case and derive its basic properties. Subsequently, the results concerning the entropy are used to introduce the concept of mutual information of intuitionistic fuzzy experiments. In addition, we define the Kullback-Leibler divergenceand its conditional versionin the intuitionistic fuzzy caseand examine the algebraic properties of the proposed quantities. The results are explained with several examples to illustrate the developed theory.

Keywords

Intuitionistic fuzzy set IF-partition Shannon entropy mutual information Kullback-Leibler divergence

1 Introduction

It is well–known that a classical Cantor’s set corresponds to an indicator function, of which values are only taken to be 0 and 1. The concept of fuzzy set, introduced by L.A. Zadeh in 1965 [42], extends the classical set theory. Recall that a fuzzy set is a mapping f : X → [0, 1] (where f (x) is interpreted as the degree of membership of the element x ∈ X to the considered fuzzy set f). The fuzzy set theory allows mathematically describing inaccurate, vague terms and working with them. It has been applied in various fields of mathematical research and it is continually developing. In addition, the fuzzy set theory has important applications also in practice, e.g., in control engineering, data processing, management, logistics, artificial intelligence, computer science, medicine, decision theory, expert systems, logic, management science, operations research, pattern recognition, and robotics.

The theory of intuitionistic fuzzy sets, developed by K. Atanassov [1, 2], further extends both concepts by allowing the assessment of the elements by two functions: μ _A for membership and ν _A for non-membership, which belong to the real unit interval [0,1] and whose sum belongs to the same interval, as well. So, the intuitionistic fuzzy set theory is a more powerful tool to deal with vagueness. In the last two decades, many authors have paid attention to the intuitionistic fuzzy set theory that has been successfully applied in various areas such as logic programming, decision making problems, medical diagnosis, etc. Recently important applications of intuitionistic fuzzy set to artificial intelligence have appeared – intuitionistic fuzzy expert systems, intuitionistic fuzzy neural networks, intuitionistic fuzzy decision making, intuitionistic fuzzy machine learning, intuitionistic fuzzy semantic representations, etc. We refer the interested reader to the paper [21], which provides an insight on the characteristics of the highly cited intuitionistic fuzzy publications.

When planning experiments, it is important to know how much information we gain from their realization. As it is known, a measure of information is entropy, the standard approach being based on the Shannon entropy [39]. Let us remind the reader that the Shannon entropy of probability distribution P = (p ₁, p ₂, …, p _n) is the number $h (P) = \sum_{i = 1}^{n} φ (p_{i})$ , where $φ : [0, 1] \to R$ is the Shannon entropy function defined by the following equation: $φ (x) = {\begin{matrix} - x log x, & if x > 0; \\ 0, & if x = 0 . \end{matrix}$ (1)

The usual mathematical model of a random experiment in classical informationtheory [19] is a measurable partition of a probability space. Partitions are standardly defined in the context of classical Cantor’s set theory.In many cases, however, it has been shown that the partitions defined by means of fuzzy set theoryare more appropriate to solving real problems. That is why various suggestions for a generalization of the classical partitions to fuzzy partitions have been created. A fuzzy partition can serve as a mathematical model of a random experiment whose results are unclearly, vaguely defined events, the so-called fuzzy events. We refer the interested readers to (for example) [12 , 40] for some results related to the concepts of fuzzy partition and its entropy. It is known that there are many possibilities to define operations for modeling the union and intersection of fuzzy sets; an overview can be found in [11]. It should be noted that while the model studied in [23 –26] was based on Zadeh’s max-min operations (cf. [42]), in our article [27], the Łukasiewicz connectives were used to define the fuzzy set operations. Let us remind that the union of fuzzy subsets f, g of X is defined by Łukasiewicz as min(f + g, 1) . We note that in [25, 26], we have exploited our results regarding the entropy of fuzzy partitions provided in [24] to introduce the concepts of mutual information and Kullback-Leibler divergence for the fuzzy case. The notion of Kullback and Leibler divergence(often shortened to K-L divergence) was introduced in [22] as a distance measure between two probability distributions. It plays significant roles in information theory and various disciplines such as statistics, machine learning, physics, neuroscience, computer science, linguistics, etc.

We note that some suggestions for generalizing the fuzzy partitions to intuitionistic fuzzy partitions (IF-partitions) are given, for example, in [4 , 28]. Some results concerning the entropy in the intuitionistic fuzzy case can be found e.g., in [5 , 41]. The purpose of the present study is to provide a generalization of the results presented in [26] from the fuzzy case to the intuitionistic fuzzy one. Analogously as in the fuzzy case, there are many possibilities to define operations over intuitionistic fuzzy sets (see e.g., [3 , 38]). We note that while the model studied in this work is based on the Łukasiewicz connectives, the authors of the above cited papers used other connectives to define operations over intuitionistic fuzzy sets.

It has been shown that in addressing some specific problems, instead of Shannon entropy,it is more appropriate to use an approach based on the concept of logical entropy (see e.g., [8, 17]). In our article [28], the logical entropy and logical mutual information of experiments in the intuitionistic fuzzy case have been studied. We remind the reader that if P = (p ₁, p ₂, …, p _n) is a probability distribution, then the logical entropy of P (cf. [15, 16]) is defined as the number $h_{L} (P) = \sum_{i = 1}^{n} L (p_{i})$ , where $L : [0, 1] \to R$ is the logical entropy function defined by $L (x) = x (1 - x),$ (2) for every x ∈ [0, 1]. Note that while the theory of entropy for the intuitionistic fuzzy case provided in [28] is based on the concept of logical entropy function, in this paper we use an approach based on the concept of the Shannon entropy function.

The rest of the paper is structured as follows. In the following section, we provide basic definitions, notations and facts used in the article. The main results of the paper are presented in Sections 3 and 4. In Section 3, we define the Shannon–typeentropy of IF-partitions and its conditional version. It is shown that the proposed entropy measures have properties analogous to the properties of Shannon’s entropy of classical measurable partitions. Subsequently, using the results concerning the entropy of IF-partitions, the concepts of mutual information and conditional mutual information of experiments in the intuitionistic fuzzy case are introduced and basic properties of the suggested measures are proved. In Section 4, we define the K-L divergenceand its conditional version in the intuitionistic fuzzy case and examine the algebraic properties of the proposed quantities. The results are explained with several examples to illustrate the theory developed in the article. The final section provides a brief summary.

2 Basic definitions, notations and facts

In this section, we recall basic definitions, notations and facts used in the paper.

Definition 2.1. Let X be a non–empty set. By an intuitionistic fuzzy set (IF-set for short), we will mean a pair A = (μ _A, ν _A) of functions μ _A, ν _A : X → [0, 1] such that μ _A (x) + ν _A (x) ≤1, for every x ∈ X.

As already mentioned in the introduction, there are many possibilities to define operations over intuitionistic fuzzy sets. We will use the partial binary operation ⊕, and the binary operation · defined as follows. If A = (μ _A, ν _A), and B = (μ _B, ν _B) are IF-sets, then we define A ⊕ B = (μ _A + μ _B, ν _A + ν _B - 1_X), and A · B = (μ _A · μ _B, ν _A + ν _B - ν _A · ν _B). Here, 1_X denotes the constant function with the value 1; similarly, 0_X denotes the constant function with the value 0. It is obvious that if A, B are two IF-sets, then A ⊕ B is an IF-set if and only if μ _A + μ _B ≤ 1_X, and ν _A + ν _B ≥ 1_X. In the case that A ⊕ B is an IF-set, we will say that A ⊕ B exists. Put 1 = (1_X, 0_X), 0 = (0_X, 1_X). It can be verified that A · 1 = A, and A ⊕ 0 = A, for any IF-set A. For any IF-sets A, B, C, the following conditions are satisfied: (i) if A ⊕ B exists, then B ⊕ A exists, and A ⊕ B = B ⊕ A (commutativity); (ii) if (A ⊕ B) ⊕ C exists, then A ⊕ (B ⊕ C) exists, and (A ⊕ B) ⊕ C = A ⊕ (B ⊕ C) (associativity); (iii) if A ⊕ B exists, then C · A ⊕ C · B exists, and C · (A ⊕ B) = C · A ⊕ C · B (distributivity).

In the class of all IF-sets, we define the relation of partial ordering ≤ in the following way: if A = (μ _A, ν _A), and B = (μ _B, ν _B) are two IF-sets, then A ≤ B if and only if μ _A ≤ μ _B, and ν _A ≥ ν _B. It is evident that 0 ≤ A ≤ 1 for any IF-set A. Gutierrez Garcia and Rodabaugh have shown in [20] that the intuitionistic fuzzy sets ordering and topology are reduced to the ordering and topology of fuzzy sets. Another situation is in measure theory [4], where the intuitionistic fuzzy case cannot be reduced to the fuzzy one. We note that a probability theory for the intuitionistic fuzzy case has been developed in [35], see also [36].

Example 2.2. A fuzzy set f : X → [0, 1] can be regarded as an IF-set, if we put A = (f, 1_X - f). If f = I _A, where I _A is the indicator function of a set A ⊂ X, then the corresponding IF-set has the form A = (I _A, 1_X - I _A) = (I _A, I _{A
^C}). Here, A ^C denotes the complement of a set A ⊂ X. In this case A ⊕ B corresponds to the union of sets A, B ⊂ X with empty intersection, A · B to the intersection of sets A, B ⊂ X, and the relation ≤ to the inclusion of sets A, B ⊂ X.

In the paper, we shall denote by the symbol $F$ any family of IF-sets satisfying the following two conditions: (i) $0 \in F$ , $1 \in F$ ; (ii) if $A, B \in F$ , then $A \cdot B \in F$ . Any IF-set from the family $F$ is considered as an intuitionistic fuzzy event. The IF-set 0 = (0_X, 1_X) is interpreted as an impossible event; the IF-set 1 = (1_X, 0_X) as a certain event. Analogously as in [28], we define a state on the family $F$ . It plays the role of a probability measure on the family $F$ of IF-events.

Definition 2.3. A mapping $s : F \to [0, 1]$ is called a state if the following two conditions are satisfied: (i)s (1) = 1; (ii) if $A, B \in F$ such that $A \oplus B \in F$ , then s (A ⊕ B) = s (A) + s (B) .

Definition 2.4. [28] By an IF-partition of $F$ we will mean an n-tuple α = (A ₁, A ₂, …, A _n) of (not necessarily different)elements of $F$ such that $\oplus_{i = 1}^{n} A_{i} \in F$ and $s (\oplus_{i = 1}^{n} A_{i}) = 1$ .

Definition 2.5. [28] Let α = (A ₁, A ₂, …, A _k), and β = (B ₁, B ₂, …, B _l) be two IF-partitions of $F$ . We will say that the IF-partition β is a refinement of α (with respect to a states) and write α ≺ β if for each A _i ∈ α there exists a subset I (i)⊂ { 1, 2, …, l } such that s (A _i) = s (⊕ _j∈I(i) B _j), where I (i)∩ I (j) = ∅ whenever i ≠ j, and $\cup_{i = 1}^{k} I (i) = {1, 2, \dots, l} .$

Definition 2.6. [28] Let α = (A ₁, A ₂, …, A _k), and β = (B ₁, B ₂, …, B _l) be two IF-partitions of $F$ . Their join α ∨ β is defined as an r-tuple(where r = k · l) consisting of the elements A _i · B _j, i = 1, 2, …, k, j = 1, 2, …, l.

Theorem 2.7. [28] If α, β are two IF-partitions of $F$ , then α ∨ β is also an IF-partition of $F$ . Moreover, α ≺ α ∨ β and β ≺ α ∨ β

Proposition 2.8. [28] Let α = (A ₁, A ₂, …, A _n) be an IF-partition of $F$ . Then $\sum_{i = 1}^{n} s (A_{i} \cdot B) = s (B)$ , for any $B \in F$ .

Example 2.9. Consider a probability space (X, S, P), and the class $F$ of all S-measurable IF-sets, i.e., $F = {A = (μ_{A}, ν_{A}); μ_{A}, ν_{A} : X \to [0, 1]$ are S-measurable with μ _A+ ν _A ≤ 1_X }. Let c ∈ [0, 1]. It can be verified that the mapping $s : F \to [0, 1]$ defined, for any element A = (μ _A, ν _A) of $F$ , by the formula $s (A) = \int_{X} μ_{A} dP + c (1 - \int_{X} (μ_{A} + ν_{A}) dP)$ (3) is a state.

Remark 2.10. We note that any continuous state s defined on the class $F$ of all S-measurable IF-events (i.e., a state $s : F \to [0, 1]$ satisfying the condition A _n ↗ A ⇒ s (A _n) ↗ s (A)) has the form (3); for more details, see [7, 36]. The appropriate entropy theory for the case of the class $F$ of all S-measurable IF-events was provided by Ďurica in [14]. While an IF-partition considered by Ďurica in [14] is a set α ={ A ₁, A ₂, …, A _n } of S-measurable IF-setssuch that $\oplus_{i = 1}^{n} A_{i} = 1$ , the model studied in [28], as well as in this work is more general.

3 Entropy and mutual information of IF-partitions

Each IF-partition α = (A ₁, A ₂, …, A _n) of $F$ represents, from the point of view of classical probability theory, a random experiment with a finite number of results A _i, i = 1, 2, …, n (which are intuitionistic fuzzy events) with a probability distribution p _i = s (A _i), i = 1, 2, …, n. Namely, p _i ≥ 0, for i = 1, 2, …, n, and $\sum_{i = 1}^{n} p_{i} = \sum_{i = 1}^{n} s (A_{i}) = s (\oplus_{i = 1}^{n} A_{i}) = 1$ . Hence, we definethe entropy of α = (A ₁, A ₂, …, A _n) by Shannon’s formula: $H_{s} (α) = - \sum_{i = 1}^{n} s (A_{i}) log s (A_{i}) .$ (4)

If α = (A ₁, A ₂, …, A _k), and β = (B ₁, B ₂, …, B _l) are two IF-partitions of $F$ , then we define the conditional entropy of α given B _j ∈ β by: $H_{s} (α / B_{j}) = - \sum_{i = 1}^{k} s (A_{i} / B_{j}) \cdot log s (A_{i} / B_{j}),$ where $s (A_{i} / B_{j}) = {\begin{matrix} \frac{s (A_{i} \cdot B_{j})}{s (B_{j})}, & if s (B_{j}) > 0; \\ 0, & if s (B_{j}) = 0 . \end{matrix}$

The conditional entropy of α assuming a realization of the experiment β is defined by the formula: $H_{s} (α / β) = - \sum_{i = 1}^{k} \sum_{j = 1}^{l} s (A_{i} \cdot B_{j}) \cdot log \frac{s (A_{i} \cdot B_{j})}{s (B_{j})} .$

It is assumed (based on continuity arguments) that $0 \cdot log \frac{0}{a} = 0$ if a ≥ 0. The base of the logarithm can be any positive number, but as a rule one takes logarithms to the base 2. The entropy is then expressed in bits.

Remark 3.1. Evidently, the entropy H _s (α) is always non-negative, and since s (A _i · B _j) ≤ s (B _j), the conditional entropy H _s (α/β) is non-negative as well. In general, H _s (α/β) ≠ H _s (β/α). Let us consider the IF-partition α ₀ ={ 1 } representing an experiment, the result of which is a certain event. It is easy to see that H _s (α ₀) =0, and H _s (α/α ₀) = H _s (α) .

Example 3.2. Consider a class $F$ of IF-events and a state $s : F \to [0, 1]$ . Let α = (A ₁, A ₂) be an IF-partition of $F$ with s (A ₁) = p, where p ∈ (0, 1) . Then s (A ₂) =1 - p, and the entropy of α is the number H _s (α) = - p log p - (1 - p) log(1 - p). If we set p = 0.5, then H _s (α) =1 bit.

The entropy and the conditional entropyof IF-partitions satisfy properties analogous to the properties of Shannon’s entropy of classical measurable partitions as shown by the succeeding theorems. The proof of the following theorem is based on the property stated in Lemma 3.3.

Lemma 3.3. Let β = (B ₁, B ₂, …, B _q), and γ = (C ₁, C ₂, …, C _r) be IF-partitions of $F$ such that β ≺ γ. Then, for every $A \in F$ and j = 1, 2, …, q, it holds that s (A · (⊕ _k∈I(j) C _k)) = s (A · B _j), where I (j)⊂ { 1, 2, …, r } such that s (B _j) = s (⊕ _k∈I(j) C _k), for j = 1, 2, …, q, I (j)∩ I (k) = ∅, for j ≠ k, and $\cup_{j = 1}^{q} I (j) = {1, 2, \dots, r} .$

Proof. Let us suppose that the assertion is not valid. This means that there exists j ₀∈ { 1, 2, …, q } such that $s (A \cdot (\oplus_{k \in I (j_{0})} C_{k})) < s (A \cdot B_{j_{0}}),$ or $s (A \cdot (\oplus_{k \in I (j_{0})} C_{k})) > s (A \cdot B_{j_{0}}) .$

Then: $\sum_{j = 1}^{q} s (A \cdot (\oplus_{k \in I (j)} C_{k})) < \sum_{j = 1}^{q} s (A \cdot B_{j}),$ or $\sum_{j = 1}^{q} s (A \cdot (\oplus_{k \in I (j)} C_{k})) > \sum_{j = 1}^{q} s (A \cdot B_{j}) .$

But this conclusion is a contradiction, because by Proposition 2.8 we have: $\begin{matrix} \sum_{j = 1}^{q} s (A \cdot (\oplus_{k \in I (j)} C_{k})) \\ = \sum_{j = 1}^{q} s (\oplus_{k \in I (j)} (A \cdot C_{k})) \\ = \sum_{j = 1}^{q} \sum_{k \in I (j)} s (A \cdot C_{k}) \\ = \sum_{k = 1}^{r} s (A \cdot C_{k}) = s (A), \end{matrix}$ and $\sum_{j = 1}^{q} s (A \cdot B_{j}) = s (A) .$

□

Theorem 3.4. Let α, β, γ be IF-partitions of $F$ such that β ≺ γ. Then H _s (α/γ) ≤ H _s (α/β) .

Proof. Let us suppose that α = (A ₁, A ₂, …, A _p), β = (B ₁, B ₂, …, B _q), γ = (C ₁, C ₂, …, C _r), β ≺ γ. Then for any B _j ∈ β there existsa subset I (j)⊂ { 1, 2, …, r } such that s (B _j) = ∑_k∈I(j) s (C _k), where I (j)∩ I (k) = ∅, for j ≠ k, and $\cup_{j = 1}^{q} I (j) = {1, \dots, r}$ . Put (for fixed j such that $s (B_{j}) > 0) c_{k} = \frac{s (C_{k})}{s (B_{j})}$ . Then $\sum_{k \in I (j)} c_{k} = \frac{1}{s (B_{j})} \sum_{k \in I (j)} s (C_{k}) = \frac{1}{s (B_{j})} s (B_{j}) = 1$ .

Let usconsider the function $φ : [0, \infty) \to R$ defined by Equation (1). Since the function φ is concave, we have: $\begin{matrix} \sum_{k \in I (j)} c_{k} φ (x_{k}) \leq φ (\sum_{k \in I (j)} c_{k} x_{k}), \\ for every x_{k} \in [0, 1] . \end{matrix}$

Therefore, using Lemma 3.3, we can write: $\begin{matrix} H_{s} (α / γ) \\ = - \sum_{i = 1}^{p} \sum_{k = 1}^{r} s (A_{i} \cdot C_{k}) \cdot log \frac{s (A_{i} \cdot C_{k})}{s (C_{k})} \\ = \sum_{i = 1}^{p} \sum_{k = 1}^{r} s (C_{k}) \cdot φ (s (A_{i} / C_{k})) \\ = \sum_{i = 1}^{p} \sum_{j : s (B_{j}) > 0} s (B_{j}) \sum_{k \in I (j)} \frac{s (C_{k})}{s (B_{j})} \cdot φ (s (A_{i} / C_{k})) \\ \leq \sum_{i = 1}^{p} \sum_{j : s (B_{j}) > 0} s (B_{j}) φ (\sum_{k \in I (j)} \frac{s (A_{i} \cdot C_{k})}{s (B_{j})}) \\ = \sum_{i = 1}^{p} \sum_{j : s (B_{j}) > 0} s (B_{j}) φ (\frac{s (A_{i} \cdot (\oplus_{k \in I (j)} C_{k}))}{s (B_{j})}) \\ = \sum_{i = 1}^{p} \sum_{j : s (B_{j}) > 0} s (B_{j}) φ (\frac{s (A_{i} \cdot B_{j})}{s (B_{j})}) \\ = \sum_{i = 1}^{p} \sum_{j = 1}^{q} s (B_{j}) φ (s (A_{i} / B_{j})) \\ = - \sum_{i = 1}^{p} \sum_{j = 1}^{q} s (A_{i} \cdot B_{j}) \cdot log \frac{s (A_{i} \cdot B_{j})}{s (B_{j})} \\ = H_{s} (α / β) . \end{matrix}$

□

Theorem 3.5. For every IF-partitions α, β, γ of $F$ , we have: $H_{s} (α \lor β / γ) = H_{s} (β / γ) + H_{s} (α / β \lor γ) .$

Proof. Let α = (A ₁, A ₂, …, A _p), β = (B ₁, B ₂, …, B _q), γ = (C ₁, C ₂, …, C _r). It is not hard to verify that the function φ defined by Equation (1) satisfies, for every x, y ∈ [0, ∞), the condition $φ (x \cdot y) = x \cdot φ (y) + y \cdot φ (x) .$ (5)

Using the identity (5), we get: $\begin{matrix} H_{s} (α \lor β / γ) \\ = - \sum_{i, j, k} s (A_{i} \cdot B_{j} \cdot C_{k}) \cdot log \frac{s (A_{i} \cdot B_{j} \cdot C_{k})}{s (C_{k})} \\ = \sum_{i, j, k} s (C_{k}) \cdot φ (s (A_{i} / B_{j} \cdot C_{k}) \cdot s (B_{j} / C_{k})) \\ = \sum_{i, j, k} s (C_{k}) \cdot (s (A_{i} / B_{j} \cdot C_{k}) \cdot φ (s (B_{j} / C_{k})) \\ + s (B_{j} / C_{k}) \cdot φ (s (A_{i} / B_{j} \cdot C_{k}))) \\ = - \sum_{j, k} s (B_{j} \cdot C_{k}) log \frac{s (B_{j} \cdot C_{k})}{s (C_{k})} \\ - \sum_{i, j, k} s (A_{i} \cdot B_{j} \cdot C_{k}) \cdot log \frac{s (A_{i} \cdot B_{j} \cdot C_{k})}{s (B_{j} \cdot C_{k})} \\ = H_{s} (β / γ) + H_{s} (α / β \lor γ) . \end{matrix}$

□

Remark 3.6. Since α ∨ β = β ∨ α, we have also: $H_{s} (α \lor β / γ) = H_{s} (α / γ) + H_{s} (β / α \lor γ) .$ (6)

Theorem 3.7. For each IF-partitions α, β of $F$ , we have: $H_{s} (α \lor β) = H_{s} (α) + H_{s} (β / α) .$ (7)

Proof. In Equation (6) it suffices to put γ ={ 1 }. □

Remark 3.8. Using the principle of mathematical induction, we get the following generalization of Equation(6): $\begin{matrix} H_{s} (\lor_{k = 1}^{n} α_{k} / γ) = H_{s} (α_{1} / γ) \\ + \sum_{i = 2}^{n} H_{s} (α_{i} / (\lor_{k = 1}^{i - 1} α_{k}) \lor γ), \end{matrix}$

for every IF-partitions α ₁, α ₂, …, α _n, γ of $F$ . If we put γ ={ 1 }, we get the following chain rule for entropyof IF-partitions: $H_{s} (\lor_{k = 1}^{n} α_{k}) = H_{s} (α_{1}) + \sum_{i = 2}^{n} H_{s} (α_{i} / \lor_{k = 1}^{i - 1} α_{k}) .$ (8)

Theorem 3.9. For every IF-partitions α, β, γ of $F$ , we have: H _s (α ∨ β/γ) ≤ H _s (α/γ) + H _s (β/γ) .

Proof. The inequality is a consequence of Theorems 3.5, 2.7 and 3.4. □

Example 3.10. Consider the following class $F$ of Borel measurable IF-events: $F = {A = (μ_{A}, ν_{A}); μ_{A}, ν_{A} : [0, 1] \to [0, 1]$ are Borel measurable with μ _A+ ν _A ≤ 1_X } and define a state $s : F \to [0, 1]$ by the formula: $s (A) = \int_{0}^{1} μ_{A} dx + 1 - \int_{0}^{1} (μ_{A} + ν_{A}) dx,$ (9) for every A = (μ _A, v _A) ∈ F. Further, we put A ₁ = (0 . 2_X, 0 .5_X), and A ₂ = (0 . 3_X, 0 .5_X). Since A ₁ ⊕ A ₂ = (0 . 5_X, 0_X) (therefore, $A_{1} \oplus A_{2} \in F)$ , and $s (A_{1} \oplus A_{2}) = \int_{0}^{1} 0.5 dx + 1 - \int_{0}^{1} 0.5 dx = 1$ , the pair α = (A ₁, A ₂) is an IF-partition. By simple calculations, we get that s (A ₁) = s (A ₂) =0.5, and the entropy H _s (α) =1 bit. In addition, put β = (B ₁, B ₂), where B ₁ = (0 . 4_X, 0 .6_X), B ₂ = (0 . 5_X, 0 .4_X) . The pair (B ₁, B ₂) is an IF-partition with s (B ₁) =0.4, s (B ₂) =0.6, and the entropy H _s (β) =0.97095 bit. The join of IF-partitions α and β is the quadruple α ∨ β = (A ₁ · B ₁, A ₁ · B ₂, A ₂ · B ₁, A ₂ · B ₂), where s (A ₁ · B ₁) =0.2, s (A ₁ · B ₂) =0.3, s (A ₂ · B ₁) =0.2, s (A ₂ · B ₂) =0.3. The entropy of α ∨ β is the number H _s (α ∨ β) =1.97095 bits.

The conditional entropy of α assuming a realization of the experiment β is the number: $\begin{matrix} H_{s} (α / β) = - (0.2 \cdot log \frac{0.2}{0.4} + 0.3 \cdot log \frac{0.3}{0.6} \\ + 0.2 \cdot log \frac{0.2}{0.4} + 0.3 \cdot log \frac{0.3}{0.6}) = 1 bit . \end{matrix}$

Now, it is possible to verify that the equality H _s (α ∨ β) = H _s (β) + H _s (α/β) is fulfilled.

In the following part, using the previous results, we define the notions of mutual information and conditional mutual information of IF-partitions and derive basic properties of these measures.

Definition 3.11. Let α, β be two IF-partitions of $F$ . Then we define information about α in β by the formula: $I_{s} (α, β) = H_{s} (α) - H_{s} (α / β) .$ (10)

Remark 3.12. Equations (10) and (7) imply: $I_{s} (α, β) = H_{s} (α) + H_{s} (β) - H_{s} (α \lor β) .$ (11)

Evidently, I _s (α, β) = I _s (β, α), i.e., the information measure I _s is symmetric. This means that information about α in β is equal to information about β in α. For this reason, the value I _s (α, β) is said to be mutual information of IF-partitions α, β .

Proposition 3.13. If α = (A ₁, A ₂, …, A _k), and β = (B ₁, B ₂, …, B _l) are two IF-partitions of $F$ , then $I_{s} (α, β) = \sum_{i = 1}^{k} \sum_{j = 1}^{l} s (A_{i} \cdot B_{j}) \cdot log \frac{s (A_{i} \cdot B_{j})}{s (A_{i}) \cdot s (B_{j})} .$

Proof. Since by Proposition 2.8, it holds that $s (A_{i}) = \sum_{j = 1}^{l} s (A_{i} \cdot B_{j})$ , for i = 1, 2, …, k, we get: $\begin{matrix} I_{s} (α, β) \\ = H_{s} (α) - H_{s} (α / β) \\ = - \sum_{i = 1}^{k} s (A_{i}) \cdot log s (A_{i}) \\ + \sum_{i, j} s (A_{i} \cdot B_{j}) \cdot log \frac{s (A_{i} \cdot B_{j})}{s (B_{j})} \\ = \sum_{i = 1}^{k} \sum_{j = 1}^{l} s (A_{i} \cdot B_{j}) \cdot [log \frac{s (A_{i} \cdot B_{j})}{s (B_{j})} - log s (A_{i})] \\ = \sum_{i = 1}^{k} \sum_{j = 1}^{l} s (A_{i} \cdot B_{j}) \cdot log \frac{s (A_{i} \cdot B_{j})}{s (A_{i}) \cdot s (B_{j})} . \end{matrix}$

□

Theorem 3.14. Let α, β be IF-partitions of $F$ . Then I _s (α, β) ≥ 0 with the equality if and only if the IF-partitions α, β are statistically independentwith respect to s, i.e., s (A · B) = s (A) · s (B) , for every A ∈ α, and for every B ∈ β.

Proof. Let us assume that α = (A ₁, A ₂, …, A _k), and (B ₁, B ₂, …, B _l). Put δ ={ (i, j) ; s (A _i · B _j) >0 }. Then according to the inequality ln x ≤ x - 1, that holds for all real numbers x > 0 (with the equality if and only if x = 1), for every (i, j) ∈ δ, we get: $\begin{matrix} s (A_{i} \cdot B_{j}) \cdot log \frac{s (A_{i}) \cdot s (B_{j})}{s (A_{i} \cdot B_{j})} \\ = s (A_{i} \cdot B_{j}) \cdot ln 2 \cdot ln \frac{s (A_{i}) \cdot s (B_{j})}{s (A_{i} \cdot B_{j})} \\ \leq ln 2 [s (A_{i}) \cdot s (B_{j}) - s (A_{i} \cdot B_{j})] . \end{matrix}$

The equality holds if and only if $\frac{s (A_{i}) \cdot s (B_{j})}{s (A_{i} \cdot B_{j})} = 1$ , i.e., when s (A _i · B _j) = s (A _i) · s (B _j). Therefore, by Propositions 3.13 and 2.8, we obtain: $\begin{matrix} - I_{s} (α, β) = \sum_{i = 1}^{k} \sum_{j = 1}^{l} s (A_{i} \cdot B_{j}) \cdot log \frac{s (A_{i}) \cdot s (B_{j})}{s (A_{i} \cdot B_{j})} \\ = \sum_{(i, j) \in δ} s (A_{i} \cdot B_{j}) \cdot log \frac{s (A_{i}) \cdot s (B_{j})}{s (A_{i} \cdot B_{j})} \\ \leq ln 2 \sum_{(i, j) \in δ} [s (A_{i}) \cdot s (B_{j}) - s (A_{i} \cdot B_{j})] \\ \leq ln 2 \sum_{i = 1}^{k} \sum_{j = 1}^{l} [s (A_{i}) \cdot s (B_{j}) - s (A_{i} \cdot B_{j})] \\ = ln 2 [\sum_{i = 1}^{k} s (A_{i}) \cdot \sum_{j = 1}^{l} s (B_{j}) - \sum_{i = 1}^{k} \sum_{j = 1}^{l} s (A_{i} \cdot B_{j})] \\ = ln 2 (1 \cdot 1 - \sum_{i = 1}^{k} s (A_{i})) = ln 2 (1 \cdot 1 - 1) = 0 . \end{matrix}$

It follows that I _s (α, β) ≥ 0. The equality I _s (α, β) = 0 holdsif and only if s (A _i · B _j) = s (A _i) · s (B _j), for i = 1, 2, …, k, and j = 1, 2, …, l, i.e., when the IF-partitions α, β are statistically independent with respect to s. □

As a consequence of Equation (11) and Theorem 3.14, we obtain the properties of sub-additivity and additivity of entropy IF-partitions stated in the following theorem.

Theorem 3.15. For IF-partitions α, β of $F$ , it holds H _s (α ∨ β) ≤ H _s (α) + H _s (β) with the equality if and only if α, β are statistically independent with respect to s.

In our article [28], we proved the fundamental properties of the logical entropy H _L (α) of IF-partitions. It turned out that, unlike the Shannon entropy of IF-partitions, the logical entropy of IF-partitions does not have the additivity property. It satisfies the following weaker property: if IF-partitions α, β of $F$ are statistically independent, then 1 - H _L (α ∨ β) = (1 - H _L (α)) · (1 - H _L (β)). As a consequence of Equation (10) and Theorem 3.14 we obtain the following property of entropy IF-partitions.

Theorem 3.16. For IF-partitions α, β of $F$ , it holds H _s (α/β) ≤ H _s (α) with the equality if and only if the IF- partitions α, β are statistically independent with respect to s.

Example 3.17. Consider the class $F$ of Borel measurable IF-events from Example 3.10 and the state $s : F \to [0, 1]$ defined by the formula (9). Let us calculate the mutual information I _s (α, β) of IF-partitions α, β considered in Example 3.10. It is possible to verify that the IF-partitions α, β are statistically independent with respect to s. By Equation (10), we get that I _s (α, β) = H _s (α) - H _s (α / β) = 1 -1 = 0 bit. It can be verified that the equality I _s (α, β) = H _s (α) + H _s (β) - H _s (α ∨ β) is satisfied.

Definition 3.18. Let α, β, γ be IF-partitions of $F$ . Then the conditional mutual information of α and β given γ is defined by the formula $I_{s} (α, β / γ) = H_{s} (α / γ) - H_{s} (α / β \lor γ) .$ (12)

Remark 3.19. Since γ ≺ β ∨ γ, by Theorem 3.4, we have the inequality H _s (α/γ) ≥ H _s (α/β ∨ γ). Therefore, the conditional mutual information is always non-negative. In addition, it is easy to see that I _s (α, β/γ) = I _s (β, α/γ) .

Theorem 3.20. For any IF- partitions α, β, γ of $F$ , we have I _s (α ∨ β , γ) = I _s (α, γ) + I _s (β, γ/α) .

Proof. Let us calculate: $\begin{matrix} I_{s} (α, γ) + I_{s} (β, γ / α) = I_{s} (γ, α) + I_{s} (γ, β / α) \\ = H_{s} (γ) - H_{s} (γ / α) + H_{s} (γ / α) - H_{s} (γ / α \lor β) \\ = I_{s} (γ, α \lor β) = I_{s} (α \lor β, γ) . \end{matrix}$

□

The following theorem will establishchain rulesfor mutual information of IF-partitions.

Theorem 3.21. Let α ₁, α ₂, …, α _n, β be IF- partitions of $F$ . Then, for n = 2, 3, …, the following equality holds: $\begin{matrix} I_{s} (\lor_{k = 1}^{n} α_{k}, β) \\ = I_{s} (α_{1}, β) + \sum_{i = 2}^{n} I_{s} (α_{i}, β / \lor_{k = 1}^{i - 1} α_{k}) . \end{matrix}$

Proof. The assertion follows by applying Equations (10), (8), Remark 3.8, and Definition 3.18. □

Definition 3.22. Let α, β, γ be IF-partitions of $F$ . We say that α is conditionally independent to γ given β if I _s (α, γ/β) = 0 .

Remark 3.23. Since I _s (α, γ/β) = I _s (γ, α/β), we can say that IF-partitions α and γ are conditionally independent given β.

Theorem 3.24. Let α, β, γ be IF-partitions of $F$ such that I _s (α, γ/β) = 0, i.e., α and γ are conditionally independent given β. Then we have:

I _s (α∨ β, γ) = I _s (β, γ) ;

I _s (β, γ) = I _s (γ, α)+ I _s (γ, β/α) ;

I _s (α, β/γ)≤ I _s (α, β) ;

I _s (α, β) ≥ I _s (α, γ) (data processing inequality).

Proof. The proof can be easily done in the same way as the proof of Theorem 8 in [29]. □

In the following part, we analyze a concavity of entropy H _s (α) and mutual information I _s (α, β) as functions of s. We will use the symbol $S (F)$ defined on $F$ . It is routine to prove that if $s_{1}, s_{2} \in S (F)$ , then, for every real number c ∈ [0, 1], it holds that ${cs}_{1} + (1 - c) s_{2} \in S (F) .$

Theorem 3.25. Let $s_{1}, s_{2} \in S (F),$ and α be a given IF-partition of $F$ Then, for every real number c ∈ [0, 1], the following inequality holds: $c H_{s_{1}} (α) + (1 - c) H_{s_{2}} (α) \leq H_{{cs}_{1} + (1 - c) s_{2}} (α) .$

Proof. Let α = (A ₁, A ₂, …, A _n). The function φ defined by Equation (1) is concave, therefore, we get: $\begin{matrix} c H_{s_{1}} (α) + (1 - c) H_{s_{2}} (α) \\ = - \sum_{i = 1}^{n} s_{1} (A_{i}) log s_{1} (A_{i}) \\ - (1 - c) \sum_{i = 1}^{n} s_{2} (A_{i}) log s_{2} (A_{i}) \\ = \sum_{i = 1}^{n} (c φ (s_{1} (A_{i})) + (1 - c) φ (s_{2} (A_{i}))) \\ \leq \sum_{i = 1}^{n} φ (c s_{1} (A_{i}) + (1 - c) s_{2} (A_{i})) \\ = \sum_{i = 1}^{n} φ ((c s_{1} + (1 - c) s_{2}) (A_{i})) \\ = H_{c s_{1} + (1 - c) s_{2}} (α) . \end{matrix}$

The result proves that the function s → H _s (α) is concave on the class $S (F)$ .□

Remark 3.26. If α = (A ₁, A ₂, …, A _k), and β = (B ₁, B ₂, …, B _l) are two IF-partitions of $F$ , then we can write $H_{s} (β / α) = \sum_{i = 1}^{k} \sum_{j = 1}^{l} s (A_{i}) \cdot φ (s (B_{j} / A_{i}))$ , i.e., there exists c _ij = φ (s (B _j/A _i)) ≥0 such that $H_{s} (β / α) = \sum_{i = 1}^{k} \sum_{j = 1}^{l} s (A_{i}) \cdot c_{ij} .$

Put $K (F) = {s \in S (F); H_{s} (β / α) = \sum_{i = 1}^{k} \sum_{j = 1}^{l} s (A_{i}) \cdot c_{ij}} .$

Theorem 3.27. Let α, β be given IF-partitionsof $F$ . Then the mutual information s → I _s (α, β) is a concave function on the class $K (F) .$

Proof. Let s be a state from the class $K (F)$ . By definition I _s (α, β) = H _s (β) - H _s (β / α), thus the mutual information I _s (α, β) is the sum of two concave functions on the class $K (F)$ , therefore it is concave on the class $K (F)$ .□

4 K-L Divergence in the intuitionistic fuzzy case

In this section we introduce the concept of K-L divergence for the intuitionistic fuzzy case and we examine the properties of this quantity. In the proofs we use the known log sum inequality. Let a ₁, a ₂, …, a _n, and b ₁, b ₂, …, b _n be non-negative real numbers. Denote the sum $\sum_{i = 1}^{n} a_{i}$ by a, and the sum $\sum_{i = 1}^{n} b_{i}$ by b. The log sum inequality states that $\sum_{i = 1}^{n} a_{i} \cdot log \frac{a_{i}}{b_{i}} \geq a \cdot log \frac{a}{b}$ with the equality if and only if a _i = c · b _i, for all i, where c is constant. It is assumed that $x \cdot log \frac{x}{0} = \infty$ if x > 0, and $0 \cdot log \frac{0}{x} = 0$ if x ≥ 0 .

Definition 4.1. Let s ₁, s ₂ be states defined on $F$ , and α = (A ₁, …, A _n) be a given IF-partitionof $F$ . Then we define the K-L divergence d _α (s ₁||s ₂) by: $d_{α} (s_{1} | | s_{2}) = \sum_{i = 1}^{n} s_{1} (A_{i}) \cdot log \frac{s_{1} (A_{i})}{s_{2} (A_{i})} .$

Example 4.2. Consider any class $F$ of IF-events and states s ₁, s ₂, s ₃ defined on $F$ . Let α = (A ₁, A ₂) be an IF-partition of $F$ with s ₁ (A ₁) = p, s ₂ (A ₁) = q, s ₃ (A ₁) = r, where p, q, r ∈ (0, 1). Then it holds s ₁ (A ₂) =1 - p, s ₂ (A ₂) =1 - q, and s ₃ (A ₂) =1 - r. Put $p = \frac{1}{2}$ , $q = \frac{1}{3}$ , $r = \frac{1}{4}$ . Simple calculations will show that: $\begin{matrix} d_{α} (s_{1} | | s_{2}) = p \cdot log \frac{p}{q} + (1 - p) \cdot log \frac{1 - p}{1 - q} \\ = \frac{1}{2} \cdot log \frac{3}{2} + \frac{1}{2} \cdot log \frac{3}{4} = 0.084963 bit . \end{matrix}$

Analogously we get that d _α (s ₁||s ₃) =0.207519 bit, and d _α (s ₂||s ₃) =0.025062 bit. Evidently, $d_{α} (s_{1} | | s_{3}) > d_{α} (s_{1} | | s_{2}) + d_{α} (s_{2} | | s_{3}) .$

The result means that the K-L divergence is not a metric in a true sense since the triangle inequality does not hold, in general.

Theorem 4.3. Let s ₁, s ₂ be states from the class $S (F)$ and α = (A ₁, A ₂, …, A _n) be any IF-partition of $F$ . Then d _α (s ₁||s ₂) ≥0 with the equality if and only if s ₁ (A _i) = s ₂ (A _i), for i = 1, 2, …, n.

Proof. The assertion follows by applying the log-sum inequality. It suffices to put a _i = s ₁(A _i) and b _i = s ₂(A _i), for i = 1, 2, …, n. Then a ₁, a ₂, …, a _n, b ₁, b ₂, …, b _n are non-negative real numbers such that $a = \sum_{i = 1}^{n} a_{i} = 1$ and $b = \sum_{i = 1}^{n} b_{i} = 1$ . Namely, $a = \sum_{i = 1}^{n} a_{i} = \sum_{i = 1}^{n} s_{1} (A_{i}) = 1$ ; analogously we obtain that $b = \sum_{i = 1}^{n} b_{i} = 1$ . Therefore, we get: $\begin{matrix} d_{α} (s_{1} | | s_{2}) = \sum_{i = 1}^{n} s_{1} (A_{i}) \cdot log \frac{s_{1} (A_{i})}{s_{2} (A_{i})} \\ = \sum_{i = 1}^{n} a_{i} \cdot log \frac{a_{i}}{b_{i}} \geq a \cdot log \frac{a}{b} = 1 \cdot log \frac{1}{1} = 0 \end{matrix}$ with the equality if and only if s ₁ (A _i) = c · s ₂ (A _i), for i = 1, 2, …, n, where c is constant. Taking the sum for all i = 1, 2, …, n, we obtain the equality $\sum_{i = 1}^{n} s_{1} (A_{i}) = c \sum_{i = 1}^{n} s_{2} (A_{i})$ , which implies that c = 1. This means that d _α (s ₁||s ₂) =0 if and only if s ₁ (A _i) = s ₂ (A _i), for i = 1, 2, …, n.□

We note that in the classical theory, the inequality from the previous theorem is known as Gibb’s inequality. In the following text, the symbol card α denotes the cardinality of the set α .

Theorem 4.4. Let s be a state defined on $F$ , and α = (A ₁, A ₂, …, A _n) be an IF-partition of $F$ . Then it holds $H_{s} (α) = log card α - d_{α} (s | | t),$ where $t : F \to [0, 1]$ is a state uniform over α, i.e., $t (A_{i}) = \frac{1}{n}$ , for i = 1, 2, …, n .

Proof. Let us calculate: $\begin{matrix} d_{α} (s | | t) & = & \sum_{i = 1}^{n} s (A_{i}) \cdot log \frac{s (A_{i})}{t (A_{i})} \\ = & \sum_{i = 1}^{n} s (A_{i}) \cdot log \frac{s (A_{i})}{n^{- 1}} \\ = & \sum_{i = 1}^{n} s (A_{i}) \cdot (log s (A_{i}) + log n) \\ = & log card α - H_{s} (α) . \end{matrix}$

□

By combining the previous two assertions, we obtain the following property of entropy of IF-partitions.

Corollary 4.5. For arbitrary IF-partition α of $F$ , we have H _s (α) ≤ log card α with the equality if and only if the state s is uniform over α.

Proof. Let α = (A ₁, A ₂, …, A _n) and $t : F \to [0, 1]$ be a state uniform over α. According to Theorem 4.4, we have d _α (s||t) = log card α - H _s (α) . By Theorem 4.3 it holds d _α (s||t) ≥0, and therefore, we get the inequality H _s (α) ≤ log card α . Moreover, by Theorem 4.3 d _α (s||t) =0 if and only if s (A _i) = t (A _i), for i = 1, 2, …, n. Thus the equality H _s (α) = log card α holds if and only if s is uniform over α .□

Theorem 4.6. The K-L divergence d _α (s ₁||s ₂) is convex in the pair (s ₁, s ₂), i.e., if $(s_{1}^{'}, s_{2}^{'})$ , $(s_{1}^{″}, s_{2}^{″})$ are pairs of states defined on $F$ , then, for every real number c ∈ [0, 1], it holds: $\begin{matrix} d_{α} (c {s^{'}}_{1} + (1 - c) s_{1}^{''} | | c s_{2}^{'} + (1 - c) s_{2}^{''}) \\ \leq c d_{α} (s_{1}^{'} | | s_{2}^{'}) + (1 - c) d_{α} (s_{1}^{''} | | s_{2}^{''}) \end{matrix}$

Proof. Assume that α = (A ₁, A ₂, …, A _n and fix i ∈ { 1, 2, …, n } . Put $a_{1} = c s_{1}^{'} (A_{i})$ , $a_{2} = (1 - c) s_{1}^{″} (A_{i})$ , $b_{1} = c s_{2}^{'} (A_{i})$ , $b_{2} = (1 - c) s_{2}^{″} (A_{i})$ , in the log-sum inequality. Then: $\begin{matrix} (c s_{1}^{'} (A_{i}) + (1 - c) s_{1}^{''} (A_{i})) \cdot \log \frac{c {s^{'}}_{1} (A_{i}) + (1 - c) s_{1}^{''} (A_{i})}{c s_{2}^{'} (A_{i}) + (1 - c) s_{2}^{''} (A_{i})} \\ \leq c s_{1}^{'} (A_{i}) \log \frac{c s_{1}' (A_{i})}{c s_{2}' (A_{i})} + (1 - c) s_{1}^{''} (A_{i}) \log \frac{(1 - c) s_{1}^{''} (A_{i})}{(1 - c) s_{2}^{''} (A_{i})} . \end{matrix}$

Summing these inequalities over i = 1, 2, …, n, we obtain the assertion.□

Finally, we define the conditional K-L divergence and, using this notion, we formulate the chain rule for the K-L divergence in the intuitionistic fuzzy case.

Definition 4.7. Let s ₁, s ₂ be states defined on $F$ , and α = (A ₁, A ₂, …, A _k), β = (B ₁, B ₂, …, B _l) be two IF-partitions of $F$ . Then we define the conditionalK-L divergence d _β/α (s ₁||s ₂) by: $\begin{matrix} d_{β / α} (s_{1} | | s_{2}) \\ = \sum_{i = 1}^{k} s_{1} (A_{i}) \sum_{j = 1}^{l} s_{1} (B_{j} / A_{i}) \cdot log \frac{s_{1} (B_{j} / A_{i})}{s_{2} (B_{j} / A_{i})} . \end{matrix}$

Theorem 4.8. If α, β are two IF-partitions of $F$ , then: $d_{α \lor β} (s_{1} | | s_{2}) = d_{α} (s_{1} | | s_{2}) + d_{β / α} (s_{1} | | s_{2}) .$

Proof. Let α = (A ₁, A ₂, …, A _k), β = (B ₁, B ₂, …, B _l). Since by Proposition 2.8, it holds $s_{1} (A_{i}) = \sum_{j = 1}^{l} s_{1} (A_{i} \cdot B_{j}),$

For i = 1, 2, …, k, we get: $\begin{matrix} d_{α} (s_{1} ∥ s_{2}) + d_{β / α} (s_{1} ∥ s_{2}) \\ = \sum_{i = 1}^{k} s_{1} (A_{i}) \cdot log \frac{s_{1} (A_{i})}{s_{2} (A_{i})} \\ + \sum_{i = 1}^{k} s_{1} (A_{i}) \sum_{j = 1}^{l} s_{1} (B_{j} / A_{i}) \cdot log \frac{s_{1} (B_{j} / A_{i})}{s_{2} (B_{j} / A_{i})} \\ = \sum_{i = 1}^{k} \sum_{j = 1}^{l} s_{1} (A_{i} \cdot B_{j}) \cdot log \frac{s_{1} (A_{i})}{s_{2} (A_{i})} \\ + \sum_{i = 1}^{k} \sum_{j = 1}^{l} s_{1} (A_{i} \cdot B_{j}) \cdot log \frac{s_{1} (B_{j} / A_{i})}{s_{2} (B_{j} / A_{i})} \\ = \sum_{i = 1}^{k} \sum_{j = 1}^{l} s_{1} (A_{i} \cdot B_{j}) \cdot (log \frac{s_{1} (A_{i})}{s_{2} (A_{i})} + log \frac{s_{1} (B_{j} / A_{i})}{s_{2} (B_{j} / A_{i})}) \\ = \sum_{i = 1}^{k} \sum_{j = 1}^{l} s_{1} (A_{i} \cdot B_{j}) \cdot log \frac{s_{1} (A_{i}) s_{1} (B_{j} / A_{i})}{s_{2} (A_{i}) s_{2} (B_{j} / A_{i})} \\ = \sum_{i = 1}^{k} \sum_{j = 1}^{l} s_{1} (A_{i} \cdot B_{j}) \cdot log \frac{s_{1} (A_{i} \cdot B_{j})}{s_{2} (A_{i} \cdot B_{j})} = d_{α \lor β} (s_{1} ∥ s_{2}) . \end{matrix}$

We usedthe implication s ₁ (A _i) =0 ⇒ s ₁ (A _i · B _j) =0 that follows from the equality $s_{1} (A_{i}) = \sum_{j = 1}^{l} s_{1} (A_{i} \cdot B_{j})$ . □

5 Conclusion

In the paper, we introduced the concept of Shannon-type entropy of IF-partitions and its conditional version. We showed that the suggested entropy measures have properties analogous to properties of Shannon entropy in the classical case. Specifically, it was shown that the Shannon entropy of IF-partitions satisfies the properties of additivity and sub-additivity.Subsequently, the concept of entropy of IF-partitions was exploited to define the mutual information of experiments in the intuitionistic fuzzy case. We proved basic properties of this quantity, inter alia, the data processing inequality for conditionally independent IF-partitions. The proposed measures can be used whenever we need to know the amount of information obtained by realization of experiments whose results are intuitionistic fuzzy events.

In the final section, we introduced the concept of K-L divergence in the intuitionistic fuzzy case and we examined the properties of this distance measure. In particular, a convexity of K-L divergence in the intuitionistic fuzzy case was proved and the relationship between K-L divergence and entropy of IF-partitions was studied. The chain rules for entropy, mutual information and K-L divergence in the intuitionistic fuzzy case were also derived. To illustrate the results, we have provided several numerical examples.

Footnotes

Acknowledgments

The first author thanks the editor and the referees for their valuable comments and suggestions. It is my sad duty to announce that Prof. RNDr. Beloslav Riečan, DrSc., Dr.h.c. passed away unexpectedly on August 13, 2018, before this article was published. Rest in peace, our beloved friend, co-worker, and teacher.

References

Atanassov

, Intuitionistic fuzzy sets, Fuzzy Sets Syst 20 (1986), 87–96.

Atanassov

, Intuitionistic Fuzzy Sets: Theory and Applications, Physica Verlag: New York, NY, USA, 1999.

Atanassov

, Riečan

, On two operations over intuitionistic fuzzy sets, J Appl Math Stat Inform 2 (2006), 145–148.

Ban

, Intuitionistic Fuzzy Measures: Theory and Applications, Nova Science Publishers: New York, NY, USA, 2006.

Ban

, Measurable entropy of intuitionistic fuzzy dynamical system, Notes Intuit. Fuzzy Sets 6 (2000), 35–47.

Burillo

, Bustince

, Entropy on intuitionistic fuzzy sets and on interval-valued fuzzy sets, Fuzzy Sets Syst 78 (1996), 305–316.

Ciungu

, Riečan

, Representation theorem of probabilities on IFS-events, Inf Sci 180 (2010), 793–798.

Tamir

, Cohen

, Logical Entropy for Quantum States, arXiv:1412.0616v2 [quant-ph] (2014). Available online: http://arxiv.org/pdf/1412.0616.pdf

Čunderlíková

, Intuitionistic fuzzy partition, Notes Intuit Fuzzy Sets 23 (2017), 44–53.

10.

S.K.

, Biswas

, Roy

A.R.

, Some operations on intuitionistic fuzzy sets, Fuzzy Sets Syst 114 (2000), 477–484.

11.

Dubois

, Prade

, A review of fuzzy set aggregation connectives, Inf Sci 36 (1985), 85–121.

12.

Dumitrescu

, Fuzzy partitions with the connectives T _∞, S _∞ , Fuzzy Sets Syst 47 (1992), 193–195.

13.

Dumitrescu

, Fuzzy measures and entropy of fuzzy partitions, J Math An and Appl 176 (1993), 359–373.

14.

Ďurica

, Entropy on IF-events, Notes Intuit Fuzzy Sets 13 (2007), 30–40.

15.

Ellerman

, An introduction to logical entropy and its relation to shannon entropy, Int J Seman Comput 7 (2013), 121–145.

16.

Ellerman

, Logical information theory: New foundations for information theory, Log J IGPL 25 (2017), 806–835.

17.

Ellerman

, Logical entropy: Introduction to classical and quantum logical information theory, Entropy 20 (2018), 679.

18.

Farnoosh

, Rahimi

, Kumar

, Removing noise in a digital image using a new entropy method based on intuitionistic fuzzy sets, Proceedings of the International Conference on Fuzzy Systems, Vancouver, BC, Canada, 2016; Institute of Electrical and Electronics Engineers Inc.: New York, NY, USA, 2016; pp. 1328–1332.

19.

Gray

R.M.

, Entropy and Information Theory, Springer: Berlin/Heidelberg, Germany, 2009.

20.

Gutierrez Garcia

and Rodabaugh

S.E.

, Order-theoretic, topological, categorical redundancies of interval-valued sets, grey sets, vague sets, interval-valued “intuitionistic” sets, “intuitionistic” fuzzy sets and topologies, Fuzzy Sets Syst 156 (2005), 445–484.

21.

, Wu

, Global research trends of intuitionistic fuzzy set: A bibliometric analysis, J Intelligent Systems (Published Online: 2017-09-11), doi: 10.1515/jisys-2017-0240

22.

Kullback

, Leibler

R.A.

, On information and sufficiency, Ann Math Stat 22 (1951), 79–86.

23.

Markechová

, The entropy of fuzzy dynamical systems and generators, Fuzzy Sets Syst 48 (1992), 351–363.

24.

Markechová

, Entropy of complete fuzzy partitions, Math Slovaca 43 (1993), 1–10.

25.

Markechová

, Entropy and mutual information of experiments in the fuzzy case, Neural Netw World 23 (2013), 339–349.

26.

Markechová

, Kullback-leibler divergence and mutual information of experiments in the fuzzy case, Axioms 6(1) (2017). doi: 10.3390/axioms6010005

27.

Markechová

, Riečan

, Entropy of fuzzy partitions and entropy of fuzzy dynamical systems, Entropy 18(1) (2016). doi: 10.3390/e18010019

28.

Markechová

, Riečan

, Logical entropy and logical mutual information of experiments in the intuitionistic fuzzy case, Entropy 19(8) (2017). doi: 10.3390/e19080429

29.

Markechová

, Riečan

, Kullback-leibler divergence and mutual information of partitions in product MV algebras, Entropy 19(6) (2017). doi: 10.3390/e19060267

30.

Mesiar

, Reusch

, Thiele

, Fuzzy equivalence relations and fuzzy partition, J Mult Valued Log Soft Comput 12 (2006), 167–181.

31.

Mesiar

, Rybárik

, Entropy of fuzzy partitions—A general model, Fuzzy Sets Syst 99 (1998), 73–79.

32.

Montes

, et al., Fuzzy delta-epsilon partitions, Inf Sci 152 (2003), 267–285.

33.

Piasecki

, Fuzzy partitions of sets, Busefal 25 (1986), 52–60.

34.

Rahimi

, Riazi

, On local entropy of fuzzy partitions, Fuzzy Sets Syst 234 (2014), 97–108.

35.

Riečan

, Probability theory on IF events. In Algebraic and Proof-Theoretic Aspects of Non-Classical Logics, Papers in Honor of Daniele Mundici on the Occasion of his 60th Birthday, Lecture Notes in Computer Science, Springer: New York, NY, USA, 2007, pp. 290–308.

36.

Riečan

, On a problem of Radko Mesiar: General form of IF-probabilities, Fuzzy Sets Syst 152 (2006), 1485–1490.

37.

Riečan

and Atanassov

, Some properties of operations conjuction and disjunction from Łukasiewicz type over intuitionistic fuzzy sets Part 1, Notes Intuit Fuzzy Sets 20(3) (2014), 1–5.

38.

Riečan

and Atanassov

, Some properties of operations conjuction and disjunction from Łukasiewicz type over intuitionistic fuzzy sets Part 2, Notes Intuit Fuzzy Sets 20(4) (2014), 1–6.

39.

Shannon

C.E.

, A mathematical theory of communication, Bell Syst Tech J 27 (1948), 379–423.

40.

Srivastava

, Khare

, Srivastava

Y.K.

, m-Equivalence, entropy and F-dynamical systems, Fuzzy Sets Syst 121 (2001), 275–283.

41.

Szmidt

, Kacprzyk

, Entropy for intuitionistic fuzzy sets, Fuzzy Sets Syst 118 (2001), 467–477.

42.

Zadeh

L.A.

, Fuzzy sets, Inf Control 8 (1965), 338–358.