Existence and non-existence results to a mixed anisotropic Schrödinger system in a plane

Abstract

This article focuses on the existence and non-existence of solutions for the following system of local and nonlocal type $\begin{aligned} {\begin{cases} - \partial_{x x} u + {(- Δ)}_{y}^{s_{1}} u + u - u^{2_{s_{1}} - 1} = κ α h (x, y) u^{α - 1} v^{β} & in R^{2}, \\ - \partial_{x x} v + {(- Δ)}_{y}^{s_{2}} v + v - v^{2_{s_{2}} - 1} = κ β h (x, y) u^{α} v^{β - 1} & in R^{2}, \\ u, v ⩾ 0 & in R^{2}, \end{cases} \end{aligned}$ where $s_{1}, s_{2} \in (0, 1), α$ , $β > 1$ , $α + β ⩽ min {2_{s_{1}}, 2_{s_{2}}}$ , and $2_{s_{i}} = \frac{2 (1 + s_{i})}{1 - s_{i}}$ , $i = 1, 2$ . The existence of a ground state solution entirely depends on the behaviour of the parameter $κ > 0$ and on the function h. In this article, we prove that a ground state solution exists in the subcritical case if κ is large enough and h satisfies (H). Further, if κ becomes very small, then there is no solution to our system. The study of the critical case, i.e., $s_{1} = s_{2} = s$ , $α + β = 2_{s}$ , is more complex, and the solution exists only for large κ and radial h satisfying (H1). Finally, we establish a Pohozaev identity which enables us to prove the non-existence results under some smooth assumptions on h.

Keywords

Mixed Schrödinger operator system of PDEs in plane variational methods concentration-compactness non-existence

1. .Introduction

In this article, we study the existence and non-existence of solutions for the following system associated with mixed Schrödinger operator:

\begin{aligned} {\begin{cases} - \partial_{x x} u + {(- Δ)}_{y}^{s_{1}} u + u - u^{2_{s_{1}} - 1} = κ α h (x, y) u^{α - 1} v^{β} & in R^{2}, \\ - \partial_{x x} v + {(- Δ)}_{y}^{s_{2}} v + v - v^{2_{s_{2}} - 1} = κ β h (x, y) u^{α} v^{β - 1} & in R^{2}, \\ u, v ⩾ 0 & in R^{2}, \end{cases} \end{aligned}

(1.1)

where

s_{1}, s_{2} \in (0, 1)

. The exponent

2_{s_{i}} = \frac{2 (1 + s_{i})}{1 - s_{i}}

(i = 1, 2)

is known as a critical exponent and it plays a vital role in the existence and non-existence of solutions to the system (1.1). The parameter

κ > 0

and α, β are chosen such that

\begin{aligned} α, β > 1 and α + β ⩽ min {2_{s_{1}}, 2_{s_{2}}} . \end{aligned}

($\alpha\beta$)

Throughout this paper we always assume that the function h satisfies the following condition

\begin{aligned} 0 ⩽ h \in L^{1} (R^{2}) \cap L^{\infty} (R^{2}), h is not zero . \end{aligned}

(H)

To study the motivation and the applications of the mixed Schrödinger operator

L := - \partial_{x x} + {(- Δ)}_{y}^{s} + I

in detail, we refer to the article [5] and the references therein. In 2018, Amin Esfahani et al. [5] dealt with the following problem

\begin{aligned} u + {(- Δ)}_{x}^{s} u - Δ_{y} u = f (u), (x, y) \in R^{N} \times R^{M}, \end{aligned}

(1.2)

where

N, M ⩾ 1

and the operator

{(- Δ)}_{x}^{s}, s \in (0, 1)

denotes the fractional Laplacian with respect to x-variable, which is defined (on appropriate functions) in Cauchy principle value (p.v.) sense up to a normalizing constant

C_{N, s}

\begin{aligned} (- Δ)_{x}^{s} u (x, y) = p . v . \int_{R^{N}} \frac{u (x, y) - u (z, y)}{| x - z |^{N + 2 s}} d z . \end{aligned}

The function

f \in C^{1} (R, R)

satisfies the subcritical growth assumptions without Ambrosetti–Rabinowitz type condition. The equation (1.2) admits a positive radial ground state solution (see [5, Theorem 1.3]). In 2019, Felmer and Wang [6] also considered the problem of type (1.2) and studied the qualitative properties of positive solutions under different assumptions on f. Recently, Gou et al. [7] provided a comprehensive study about the existence and non-existence of solutions to the following problem on a plane

\begin{aligned} - \partial_{x x} u + {(- Δ)}_{y}^{s} u + u = u^{p - 1} in R^{2} . \end{aligned}

(1.3)

The authors proved that the problem (1.3) has a positive ground state solution which is axially symmetric for

2 < p < 2_{s}

, and using Pohozaev identity it has only trivial solution for

p ⩾ 2_{s}

. If

κ = 0

, the system (1.1) reduces to a single equation of the form

\begin{aligned} - \partial_{x x} u + {(- Δ)}_{y}^{s} u + u = u^{2_{s} - 1} in R^{2}, \end{aligned}

(1.4)

which clearly does not possess any non-trivial solution by taking

p = 2_{s}

in (1.3). This shows that the system (1.1) has no semi-trivial solutions, i.e. solutions like

(u, 0)

(0, v)

. In this article, we are interested in the effects of κ and

h (x, y)

on the existence of fully non-trivial solutions (particularly, ground state solutions, whose definition will be given later) for the system (1.1), i.e. solutions

(u, v)

with

u \neq 0

and

v \neq 0

. Since the equation (1.3) has non-trivial solutions for

2 < p < 2_{s}

(see [7]), when

α + β < min {2_{s_{1}}, 2_{s_{2}}}

, we expect that a large κ and positive h will bring the existence of ground state solutions for (1.1). Indeed, as readers will see, our expection is right. Further, this is right when

α + β = min {2_{s_{1}}, 2_{s_{2}}} < max {2_{s_{1}}, 2_{s_{2}}}

. Situations become more complex in the critical case that

s_{1} = s_{2} = s

α + β = 2_{s}

. On the one hand, we can prove that a solution exists under a setting that κ is large and h is radial and satisfies the following assumption (H1):

\begin{aligned} 0 ⩽ h \in L^{1} (R^{2}) \cap L^{\infty} (R^{2}), h is not zero, continuous near 0 and \infty, and \\ h (0, 0) = lim_{| (x, y) | \to + \infty} h (x, y) = 0. \end{aligned}

(H1)

On the other hand, the system (1.1) has no non-trivial solutions for some h, such as h being a positive constant, and more examples of h will be given later. Our study opens a door to understand that the conditions on κ and h are crucial for the existence and non-existence of non-trivial solution for (1.1).

To state our main results, we first define the fractional Sobolev–Liouville space $H^{1, s_{i}} (R^{2}), i = 1, 2$ (see [4]). The space $H^{1, s_{i}} (R^{2})$ is a Banach space with respect to the norm

\begin{aligned} ‖ u ‖_{H^{1, s_{i}}}^{2} := ‖ u ‖_{2}^{2} + ‖ \partial_{x} u ‖_{2}^{2} + {‖ {(- Δ)}_{y}^{s_{i} / 2} u ‖}_{2}^{2}, for i = 1, 2. \end{aligned}

The product space

D = H^{1, s_{1}} (R^{2}) \times H^{1, s_{2}} (R^{2})

is endowed with the norm

\begin{aligned} {‖ (u, v) ‖}_{D}^{2} = ‖ u ‖_{H^{1, s_{1}}}^{2} + ‖ v ‖_{H^{1, s_{2}}}^{2} . \end{aligned}

The energy functional associated with the system (1.1) is defined by

J_{κ} : D \to R

and given as follows:

\begin{aligned} J_{κ} (u, v) & = \frac{1}{2} \int_{R^{2}} (| u |^{2} + | \partial_{x} u |^{2} + | {(- Δ)}_{y}^{s_{1} / 2} u |^{2}) d x d y \\ + \frac{1}{2} \int_{R^{2}} (| v |^{2} + | \partial_{x} v |^{2} + | {(- Δ)}_{y}^{s_{2} / 2} v |^{2}) d x d y \\ - \frac{1}{2_{s_{1}}} \int_{R^{2}} | u |^{2_{s_{1}}} d x d y - \frac{1}{2_{s_{2}}} \int_{R^{2}} | v |^{2_{s_{2}}} d x d y - κ \int_{R^{2}} h (x, y) | u |^{α} | v |^{β} d x d y . \end{aligned}

(1.5)

The above functional

J_{κ}

is well-defined and

C^{1}

on the product space

D

. For

(u_{0}, v_{0}) \in D

, the Fréchet derivative of

J_{κ}

(u, v) \in D

is represented as

\begin{aligned} ⟨ J_{κ}^{'} (u, v) | (u_{0}, v_{0}) ⟩ & = \int_{R^{2}} (u u_{0} + \partial_{x} u \partial_{x} u_{0} + {(- Δ)}_{y}^{s_{1} / 2} u {(- Δ)}_{y}^{s_{1} / 2} u_{0}) d x d y \\ + \int_{R^{2}} (v v_{0} + \partial_{x} v \partial_{x} v_{0} + {(- Δ)}_{y}^{s_{2} / 2} v {(- Δ)}_{y}^{s_{2} / 2} v_{0}) d x d y \\ - \int_{R^{2}} | u |^{2_{s_{1}} - 2} u u_{0} d x d y - \int_{R^{2}} | v |^{2_{s_{2}} - 2} v v_{0} d x d y \\ - κ α \int_{R^{2}} h (x, y) | u |^{α - 2} u u_{0} | v |^{β} d x d y - κ β \int_{R^{2}} h (x, y) | u |^{α} | v |^{β - 2} v v_{0} d x d y, \end{aligned}

where

J_{κ}^{^{'}} (u, v)

is the Fréchet derivative of

J_{κ}

(u, v)

, and the duality bracket between the product space

D

and its dual

D^{*}

is represented as

_{D^{*}} ⟨ \cdot, \cdot ⟩_{D}

. We just write

⟨ \cdot, \cdot ⟩

if no confusion of notation arises.

Definition 1 (Ground state solution)

A non-trivial critical point $(u_{1}, v_{1}) \in D ∖ {(0, 0)}$ of $J_{κ}$ over $D$ is said to be a ground state solution if its energy is minimal among all the non-trivial critical points i.e.

\begin{aligned} c_{κ} = J_{κ} (u_{1}, v_{1}) = min {J_{κ} (u, v) : (u, v) \in D ∖ {(0, 0)} and J_{κ}^{^{'}} (u, v) = 0} . \end{aligned}

(1.6)

As explained in Section 2 below, the functional $J_{κ}$ is not bounded from below on the product space $D$ . For this reason, we introduce the definition of Nehari manifold $N_{κ}$ associated with the functional $J_{κ}$ as

\begin{aligned} N_{κ} = {(u, v) \in D ∖ {(0, 0)} : Φ_{κ} (u, v) = 0}, \end{aligned}

where

\begin{aligned} Φ_{κ} (u, v) = ⟨ J_{κ}^{^{'}} (u, v) | (u, v) ⟩ . \end{aligned}

(1.7)

Define

{\bar{c}}_{κ} := inf_{(u, v) \in N_{κ}} J_{κ} (u, v)

. Notice that the Nehari manifold

N_{κ}

consists of all non-trivial critical points of

J_{κ}

. Hence, if

{\bar{c}}_{κ}

can be achieved by some

(u, v)

, then

(u, v)

is a ground state solution.

Proposition 2.

We have the following conclusions: (1)

${\bar{c}}_{κ}$ is non-increasing w.r.t. $κ ⩾ 0$ , and further, ${\bar{c}}_{κ} > {\bar{c}}_{κ^{'}}$ for all $κ^{'} > κ$ if ${\bar{c}}_{κ}$ can be achieved;

(2)

${\bar{c}}_{κ}$ is continuous w.r.t. $κ \in [0, \infty)$ ;

(3)

there exists $κ^{*} \in [0, \infty)$ such that

\begin{aligned} {\bar{c}}_{κ} & = min {\frac{s_{1}}{1 + s_{1}} Λ_{1}^{\frac{1 + s_{1}}{2 s_{1}}}, \frac{s_{2}}{1 + s_{2}} Λ_{2}^{\frac{1 + s_{2}}{2 s_{2}}}}, \forall κ \in [0, κ^{*}], \\ {\bar{c}}_{κ} & < min {\frac{s_{1}}{1 + s_{1}} Λ_{1}^{\frac{1 + s_{1}}{2 s_{1}}}, \frac{s_{2}}{1 + s_{2}} Λ_{2}^{\frac{1 + s_{2}}{2 s_{2}}}}, \forall κ > κ^{*}, \end{aligned}

where

Λ_{i} (i = 1, 2)

will be defined by ( 3.3 ).

Remark 3.

It is open whether $κ^{*} = 0$ or not.

We are firstly concerned with the case that $2 < α + β < min {2_{s_{1}}, 2_{s_{2}}}$ or $α + β = min {2_{s_{1}}, 2_{s_{2}}} < max {2_{s_{1}}, 2_{s_{2}}}$ .

Theorem 4.

Let $2 < α + β < min {2_{s_{1}}, 2_{s_{2}}}$ or $α + β = min {2_{s_{1}}, 2_{s_{2}}} < max {2_{s_{1}}, 2_{s_{2}}}$ . Recall that $κ^{*}$ is given by Proposition 2. Then we have: (1)

for any $κ \in (κ^{*}, \infty)$ , ${\bar{c}}_{κ}$ has a minimizer, which is a non-negative (constant sign) ground state solution to the system (1.1);

(2)

for any $κ \in [0, κ^{*})$ , ${\bar{c}}_{κ}$ has no minimizers.

Next we focus on the critical case that $s_{1} = s_{2} = s$ , $α + β = 2_{s}$ , which is more complex.

Theorem 5.

Suppose $s_{1} = s_{2} = s$ , $α + β = 2_{s}$ and that h is radial satisfying (H1). Then there is $κ^{* *}$ (which will be given by Proposition 7 ) such that for any $κ \in (κ^{* *}, \infty)$ , there exists a non-negative (constant sign) non-trivial solution to the system ( 1.1).

Remark 6.

It is still open whether the solution exists or not in the critical case for $κ > 0$ small enough.

To prove Theorem 5, we find solutions in the radial space $D_{r}$ where

\begin{aligned} D_{r} := H_{r}^{1, s} (R^{2}) \times H_{r}^{1, s} (R^{2}) = {(u, v) \in D : u and v are radially symmetric} . \end{aligned}

We further define

\begin{aligned} N_{κ, r} = {(u, v) \in D_{r} ∖ {(0, 0)} : Φ_{κ} (u, v) = 0}, \end{aligned}

where

\begin{aligned} Φ_{κ} (u, v) = ⟨ J_{κ}^{^{'}} (u, v) | (u, v) ⟩ . \end{aligned}

(1.8)

Let

{\bar{c}}_{κ, r} := inf_{(u, v) \in N_{κ, r}} J_{κ} (u, v)

, and

\begin{aligned} Λ_{r} := inf_{u \in H_{r}^{1, s} (R^{2})} \frac{‖ u ‖_{H^{1, s} (R^{2})}^{2}}{‖ u ‖_{2_{s}}^{2}} . \end{aligned}

(1.9)

Similar to Proposition 2, we have the following result.

Proposition 7.

Under the assumptions of Theorem 5, we have the following conclusions: (1)

${\bar{c}}_{κ, r}$ is non-increasing w.r.t. $κ ⩾ 0$ , and further, ${\bar{c}}_{κ, r} > {\bar{c}}_{κ^{'}, r}$ for all $κ^{'} > κ$ if ${\bar{c}}_{κ, r}$ can be achieved;

(2)

${\bar{c}}_{κ, r}$ is continuous w.r.t. $κ \in [0, \infty)$ ;

(3)

there exists $κ^{* *} \in [0, \infty)$ such that

\begin{aligned} {\bar{c}}_{κ, r} & = \frac{s}{1 + s} Λ_{r}^{\frac{1 + s}{2 s}}, \forall κ \in [0, κ^{* *}], \\ {\bar{c}}_{κ, r} & < \frac{s}{1 + s} Λ_{r}^{\frac{1 + s}{2 s}}, \forall κ > κ^{* *} . \end{aligned}

On the other hand, for some h, (1.1) has no non-trivial solutions.

Theorem 8 (Non-existence of non-trivial solutions)

Let $s_{1} = s_{2} = s$ , $α + β = 2_{s}$ . If h and κ satisfy either of the following hypothesis:

(a)
h is a positive constant and κ is any real number,
(b)
$h \in L^{\infty} (R^{2}) \cap C^{1} (R^{2})$ such that $κ x h_{x} ⩽ 0$ and $κ y h_{y} ⩽ 0$ ,
then the system (1.1) has no non-trivial solution $(u, v)$ which satisfies $x u, y u \in L^{2} (R^{2})$ .

We organize the rest of our paper as follows. The Section 2 contains some properties of the Nehari manifold and the boundedness of (PS) sequences. Further, the fractional Laplacian w.r.t y variable is defined and some useful results associated with it are demonstrated. In Section 3, the concentration-compactness tools for the mixed Schrödinger operator are developed and using this tool we prove the (PS) compactness condition of $J_{κ}$ in the subcritical case. In Section 4, some properties of ${\bar{c}}_{k}$ and the proof of Theorem 4 are given. In Section 5, we give the proof of Theorem 5. Finally, we establish the Pohozaev identity in Section 6 and using this Pohozaev identity we give the non-existence result (see Theorem 8). We conclude this section by providing some specific examples of h for which the non-trivial solution does not exist.
2. .Preliminaries

2.1. .The Nehari manifold and (PS) sequences

For any $η > 0$ and (1.5), we can write

\begin{aligned} J_{κ} (η u, η v) & = \frac{η^{2}}{2} \int_{R^{2}} (| u |^{2} + | \partial_{x} u |^{2} + | {(- Δ)}_{y}^{s_{1} / 2} u |^{2}) d x d y \\ + \frac{η^{2}}{2} \int_{R^{2}} (| v |^{2} + | \partial_{x} v |^{2} + | {(- Δ)}_{y}^{s_{2} / 2} v |^{2}) d x d y - \frac{η^{2_{s_{1}}}}{2_{s_{1}}} \int_{R^{2}} | u |^{2_{s_{1}}} d x d y \\ - \frac{η^{2_{s_{2}}}}{2_{s_{2}}} \int_{R^{2}} | v |^{2_{s_{2}}} d x d y - κ η^{α + β} \int_{R^{2}} h (x, y) | u |^{α} | v |^{β} d x d y . \end{aligned}

(2.1)

Observe that the functional

J_{κ} (η u, η v) \to - \infty

as η becomes very large. Since

J_{κ}

is not bounded from below on

D

, we try to minimize the given functional on the Nehari manifold

N_{κ}

in order to find a critical point in

D

via variational approach.

If we consider $(u, v) \in N_{κ}$ , then the following identity holds

\begin{aligned} {‖ (u, v) ‖}_{D}^{2} = ‖ u ‖_{2_{s_{1}}}^{2_{s_{1}}} + ‖ v ‖_{2_{s_{2}}}^{2_{s_{2}}} + κ (α + β) \int_{R^{2}} h (x, y) | u |^{α} | v |^{β} d x d y . \end{aligned}

(2.2)

The restriction of

J_{κ}

N_{κ}

is given as

\begin{aligned} J_{κ} |_{N_{κ}} (u, v) = \frac{s_{1}}{1 + s_{1}} ‖ u ‖_{2_{s_{1}}}^{2_{s_{1}}} + \frac{s_{2}}{1 + s_{2}} ‖ v ‖_{2_{s_{2}}}^{2_{s_{2}}} + κ (\frac{α + β - 2}{2}) \int_{R^{2}} h (x, y) | u |^{α} | v |^{β} d x d y . \end{aligned}

(2.3)

Further, if

(η u, η v) \in N_{κ}

for all

(u, v) \in D ∖ {(0, 0)}

, using the identity (2.2) we obtain an algebraic equation of the form

\begin{aligned} {‖ (u, v) ‖}_{D}^{2} = η^{2_{s_{1}} - 2} ‖ u ‖_{2_{s_{1}}}^{2_{s_{1}}} + η^{2_{s_{2}} - 2} ‖ v ‖_{2_{s_{2}}}^{2_{s_{2}}} + κ (α + β) η^{α + β - 2} \int_{R^{2}} h (x, y) | u |^{α} | v |^{β} d x d y . \end{aligned}

(2.4)

By following a careful analysis on the above algebraic equation, it has a unique positive solution. Hence, there exists a unique

η = η_{(u, v)} > 0

such that

(η u, η v) \in N_{κ}

for all

(u, v) \in D ∖ {(0, 0)}

. For any

(u, v) \in N_{κ}

, the identity (2.2) and the assumption (${\alpha \beta}$) altogether yield

\begin{aligned} J_{κ}^{^{″}} (u, v) [u, v]^{2} & = ⟨ Φ_{κ}^{^{'}} (u, v) | (u, v) ⟩ \\ = 2 {‖ (u, v) ‖}_{D}^{2} - 2_{s_{1}} ‖ u ‖_{2_{s_{1}}}^{2_{s_{1}}} - 2_{s_{2}} ‖ v ‖_{2_{s_{2}}}^{2_{s_{2}}} - κ (α + β)^{2} \int_{R^{2}} h (x, y) | u |^{α} | v |^{β} d x d y \\ = (2 - α - β) {‖ (u, v) ‖}_{D}^{2} + (α + β - 2_{s_{1}}) ‖ u ‖_{2_{s_{1}}}^{2_{s_{1}}} + (α + β - 2_{s_{2}}) ‖ v ‖_{2_{s_{2}}}^{2_{s_{2}}} < 0. \end{aligned}

(2.5)

From above, we deduce the following

\begin{aligned} J_{κ} (η_{(u, v)} u, η_{(u, v)} v) = max_{ζ > 0} J_{κ} (ζ u, ζ v) > J_{κ} (ζ u, ζ v) for 0 < ζ \neq η_{(u, v)} . \end{aligned}

(2.6)

Moreover, by the identity (2.2) there exists a

r_{κ} > 0

such that the product norm is bounded from below i.e.,

\begin{aligned} {‖ (u, v) ‖}_{D} > r_{κ} for all (u, v) \in N_{κ} . \end{aligned}

(2.7)

Using the Lagrange multiplier method, if

(u, v) \in D

is a critical point of

J_{κ}

on the Nehari manifold

N_{κ}

, then there exists a

ρ \in R

called Lagrange multiplier such that

\begin{aligned} (J_{κ} |_{N_{κ}})^{^{'}} (u, v) = J_{κ}^{^{'}} (u, v) - ρ Φ_{κ}^{^{'}} (u, v) = 0. \end{aligned}

Further calculations yields

ρ ⟨ Φ_{κ}^{^{'}} (u, v) | (u, v) ⟩ = ⟨ J_{κ}^{^{'}} (u, v) | (u, v) ⟩ = 0

. It is clear that

ρ = 0

, otherwise the inequality (2.5) does not hold and subsequently we obtain

J_{κ}^{^{'}} (u, v) = 0

. Hence, there is a one-to-one correspondence between the critical points of

J_{κ}

and the critical points of

J_{κ} |_{N_{κ}}

. The functional

J_{κ}

restricted on

N_{κ}

can also be written as

\begin{aligned} (J_{κ} |_{N_{κ}}) (u, v) = (\frac{1}{2} - \frac{1}{α + β}) {‖ (u, v) ‖}_{D}^{2} + (\frac{1}{α + β} - \frac{1}{2_{s_{1}}}) ‖ u ‖_{2_{s_{1}}}^{2_{s_{1}}} + (\frac{1}{α + β} - \frac{1}{2_{s_{2}}}) ‖ v ‖_{2_{s_{2}}}^{2_{s_{2}}} . \end{aligned}

(2.8)

Now the hypotheses (${\alpha \beta}$) and (2.7) combined with (2.8) give the following inequality

\begin{aligned} J_{κ} (u, v) > (\frac{1}{2} - \frac{1}{α + β}) r_{κ}^{2} for all (u, v) \in N_{κ} . \end{aligned}

Notice that

J_{κ}

restricted on

N_{κ}

is bounded from below. Hence, we can expect the solution of (1.1) by minimizing the energy functional

J_{κ}

on the Nehari manifold

N_{κ}

. Further, such a minimizer on

N_{κ}

is a ground state solution since

N_{κ}

consists of all non-trivial critical points of

J_{κ}

Next we prove that the (PS) sequences on $N_{κ}$ are bounded in $D$ .

Lemma 9.

If ${(u_{n}, v_{n})} \subset N_{κ}$ is a (PS) sequence for $J_{κ}$ on $N_{κ}$ at level $c \in R$ under the hypotheses (${\alpha \beta}$) and (H), then ${(u_{n}, v_{n})}$ is a bounded (PS) sequence for $J_{κ}$ in $D$ .

Proof.

Given ${(u_{n}, v_{n})} \subset N_{κ}$ is a (PS) sequence for $J_{κ}$ at level c, then by definition

\begin{aligned} J_{κ} (u_{n}, v_{n}) \to c in R as n \to \infty, i . e ., c + o_{n} (1) = J_{κ} (u_{n}, v_{n}) . \end{aligned}

(2.9)

The functional

J_{κ}

(u_{n}, v_{n})

is given as

\begin{aligned} J_{κ} (u_{n}, v_{n}) = \frac{1}{2} {‖ (u_{n}, v_{n}) ‖}_{D}^{2} - \frac{1}{2_{s_{1}}} ‖ u_{n} ‖_{2_{s_{1}}}^{2_{s_{1}}} - \frac{1}{2_{s_{2}}} ‖ v_{n} ‖_{2_{s_{2}}}^{2_{s_{2}}} - κ \int_{R^{2}} h (x, y) | u_{n} |^{α} | v_{n} |^{β} d x d y . \end{aligned}

(2.10)

Using the identity (2.2), we have

\begin{aligned} J_{κ} (u_{n}, v_{n}) & = \frac{1}{2} {‖ (u_{n}, v_{n}) ‖}_{D}^{2} - \frac{1}{2_{s_{1}}} ‖ u_{n} ‖_{2_{s_{1}}}^{2_{s_{1}}} - \frac{1}{2_{s_{2}}} ‖ v_{n} ‖_{2_{s_{2}}}^{2_{s_{2}}} - \frac{1}{α + β} ({‖ (u_{n}, v_{n}) ‖}_{D}^{2} - ‖ u_{n} ‖_{2_{s_{1}}}^{2_{s_{1}}} - ‖ v_{n} ‖_{2_{s_{2}}}^{2_{s_{2}}}) \\ = (\frac{1}{2} - \frac{1}{α + β}) {‖ (u_{n}, v_{n}) ‖}_{D}^{2} + (\frac{1}{α + β} - \frac{1}{2_{s_{1}}}) ‖ u_{n} ‖_{2_{s_{1}}}^{2_{s_{1}}} + (\frac{1}{α + β} - \frac{1}{2_{s_{2}}}) ‖ v_{n} ‖_{2_{s_{2}}}^{2_{s_{2}}} \\ c + o_{n} (1) & ⩾ (\frac{1}{2} - \frac{1}{α + β}) {‖ (u_{n}, v_{n}) ‖}_{D}^{2} . \end{aligned}

Thus, the sequence

{(u_{n}, v_{n})}

is bounded in

D

. Next, we combine (1.7) with (2.5) and (2.7) to obtain

\begin{aligned} ⟨ Φ_{κ}^{^{'}} (u_{n}, v_{n}) | (u_{n}, v_{n}) ⟩ ⩽ (2 - α - β) r_{κ}^{2} . \end{aligned}

(2.11)

By Lagrange multiplier method, we can assume the sequence of multipliers

{ω_{n}} \subset R

such that

\begin{aligned} (J_{κ} |_{N_{κ}})^{^{'}} (u_{n}, v_{n}) = J_{κ}^{^{'}} (u_{n}, v_{n}) - ω_{n} Φ_{κ}^{^{'}} (u_{n}, v_{n}) in the dual space D^{*} . \end{aligned}

(2.12)

Since

(J_{κ} |_{N_{κ}})^{^{'}} (u_{n}, v_{n})

converges to 0 as

n \to \infty

in the dual space

D^{*}

and

{(u_{n}, v_{n})}

is bounded in

D

, we obtain

\begin{aligned} ⟨ (J_{κ} |_{N_{κ}})^{^{'}} (u_{n}, v_{n}) | (u_{n}, v_{n}) ⟩ \to 0 as n \to \infty . \end{aligned}

Using

⟨ Φ_{κ}^{^{'}} (u_{n}, v_{n}) | (u_{n}, v_{n}) ⟩ < 0

, we get

ω_{n} \to 0 in R as n \to \infty

. Hence, we deduce the required result by (2.12).

In the following, we prove that the (PS) sequence in $D$ is also bounded in $D$ . Lemma 10.

If ${(u_{n}, v_{n})} \subset D$ is a (PS) sequence for $J_{κ}$ at level $c \in R$ under the hypotheses (${\alpha \beta}$) and (H), then the sequence ${(u_{n}, v_{n})}$ is bounded in $D$ .

Proof.

Given ${(u_{n}, v_{n})} \subset D$ is a (PS) sequence for $J_{κ}$ at level c, then for $n \to \infty$ we have

\begin{aligned} J_{κ} (u_{n}, v_{n}) \to c in R, \end{aligned}

(2.13)

\begin{aligned} J_{κ}^{^{'}} (u_{n}, v_{n}) \to 0 in D^{*} . \end{aligned}

(2.14)

From (2.14), we write

\begin{aligned} ⟨ J_{κ}^{^{'}} (u_{n}, v_{n}) | \frac{(u_{n}, v_{n})}{‖ (u_{n}, v_{n}) ‖_{D}} ⟩ \to 0 in R . \end{aligned}

Therefore, we have the following expression

\begin{aligned} {‖ (u_{n}, v_{n}) ‖}_{D}^{2} - ‖ u_{n} ‖_{2_{s_{1}}}^{2_{s_{1}}} - ‖ v_{n} ‖_{2_{s_{2}}}^{2_{s_{2}}} - κ (α + β) \int_{R^{2}} h (x, y) | u_{n} |^{α} | v_{n} |^{β} d x d y = o_{n} ({‖ (u_{n}, v_{n}) ‖}_{D}) as n \to \infty, \end{aligned}

and using (2.13) we get the following

\begin{aligned} \frac{1}{2} {‖ (u_{n}, v_{n}) ‖}_{D}^{2} - \frac{1}{2_{s_{1}}} ‖ u_{n} ‖_{2_{s_{1}}}^{2_{s_{1}}} - \frac{1}{2_{s_{2}}} ‖ v_{n} ‖_{2_{s_{2}}}^{2_{s_{2}}} - κ \int_{R^{2}} h (x, y) | u_{n} |^{α} | v_{n} |^{β} d x d y = c + o_{n} (1) as n \to \infty . \end{aligned}

From (2.13) and (2.14), we can write

\begin{aligned} J_{κ} (u_{n}, v_{n}) - \frac{1}{α + β} ⟨ J_{κ}^{^{'}} (u_{n}, v_{n}) | \frac{(u_{n}, v_{n})}{‖ (u_{n}, v_{n}) ‖_{D}} ⟩ = c + o_{n} (1) + o_{n} ({‖ (u_{n}, v_{n}) ‖}_{D}) as n \to \infty, \end{aligned}

which further gives the following inequality

\begin{aligned} (\frac{1}{2} - \frac{1}{α + β}) {‖ (u_{n}, v_{n}) ‖}_{D}^{2} & ⩽ (\frac{1}{2} - \frac{1}{α + β}) {‖ (u_{n}, v_{n}) ‖}_{D}^{2} + (\frac{1}{α + β} - \frac{1}{2_{s_{1}}}) ‖ u_{n} ‖_{2_{s_{1}}}^{2_{s_{1}}} \\ + (\frac{1}{α + β} - \frac{1}{2_{s_{2}}}) ‖ v_{n} ‖_{2_{s_{2}}}^{2_{s_{2}}} \\ = c + o_{n} (1) + o_{n} ({‖ (u_{n}, v_{n}) ‖}_{D}) as n \to \infty . \end{aligned}

Hence, the sequence

{(u_{n}, v_{n})}

is bounded in

D

2.2. .Fractional Laplacian operator

We denote $(- Δ)_{y}$ as the classical Laplacian operator w.r.t y variable and its fractional analogue i.e., the fractional Laplacian operator w.r.t y variable denoted by ${(- Δ)}_{y}^{s}, s \in (0, 1)$ is defined on smooth functions as

\begin{aligned} {(- Δ)}_{y}^{s} u (x, y) = C (N, s) P . V . \int_{R^{N}} \frac{u (x, y) - u (x, z)}{| y - z |^{N + 2 s}} d z, for all (x, y) \in R^{N} \times R^{M}, \end{aligned}

where

C (N, s) := {(\int_{R^{N}} \frac{1 - \cos (ζ_{1})}{| ζ |^{N + 2 s}} d ζ)}^{- 1}

is a normalizing constant and the abbreviation P.V. means the principal value sense. Since we are dealing our problem (1.1) in

R^{2}

, we choose

N = 1

and

M = 1

. So the formal definition takes the following form

\begin{aligned} {(- Δ)}_{y}^{s} u (x, y) = C (s) P . V . \int_{R} \frac{u (x, y) - u (x, z)}{| y - z |^{1 + 2 s}} d z, for all (x, y) \in R^{2}, \end{aligned}

(2.15)

where

\begin{aligned} C (s) := {(\int_{R} \frac{1 - \cos (ζ)}{| ζ |^{1 + 2 s}} d ζ)}^{- 1} . \end{aligned}

(2.16)

Due to the singularity of the kernal, the right-hand side of (2.15) is not well defined in general. In fact for

s \in (0, 1 / 2)

the integral in (2.15) is not really singular near y. In the following, we prove the analogous results of Hitchiker’s guide [3].

Lemma 11.

Let $s \in (0, 1)$ and let ${(- Δ)}_{y}^{s}$ be the fractional Laplacian operator defined by (2.15). Then, for any $u \in P (R^{2})$ (Schwartz space of rapidly decaying functions),

\begin{aligned} {(- Δ)}_{y}^{s} u (x, y) = - \frac{1}{2} C (s) \int_{R} \frac{u (x, y + z) + u (x, y - z) - 2 u (x, y)}{| z |^{1 + 2 s}} d z, \forall (x, y) \in R^{2} . \end{aligned}

(2.17)

Proof.

By choosing $t = z - y$ , we have

\begin{aligned} {(- Δ)}_{y}^{s} u (x, y) & = - C (s) P . V . \int_{R} \frac{u (x, z) - u (x, y)}{| y - z |^{1 + 2 s}} d z \\ = - C (s) P . V . \int_{R} \frac{u (x, y + t) - u (x, y)}{| t |^{1 + 2 s}} d t . \end{aligned}

(2.18)

Now by substituting

\tilde{t} = - t

in last term of the above equality, we have

\begin{aligned} P . V . \int_{R} \frac{u (x, y + t) - u (x, y)}{| t |^{1 + 2 s}} d t = P . V . \int_{R} \frac{u (x, y - \tilde{t}) - u (x, y)}{| \tilde{t} |^{1 + 2 s}} d \tilde{t} \end{aligned}

(2.19)

So after relabeling

\tilde{t}

as t

\begin{aligned} 2 P . V . \int_{R} \frac{u (x, y + t) - u (x, y)}{| t |^{1 + 2 s}} d t \\ = P . V . \int_{R} \frac{u (x, y + t) - u (x, y)}{| t |^{1 + 2 s}} d t + P . V . \int_{R} \frac{u (x, y - t) - u (x, y)}{| t |^{1 + 2 s}} d t \\ = P . V . \int_{R} \frac{u (x, y + t) + u (x, y - t) - 2 u (x, y)}{| t |^{1 + 2 s}} d t . \end{aligned}

(2.20)

Therefore, if we replace t by z in (2.18) and (2.20), we can write the fractional Laplacian operator in (2.15) as

\begin{aligned} {(- Δ)}_{y}^{s} u (x, y) = - \frac{1}{2} C (s) P . V . \int_{R} \frac{u (x, y + z) + u (x, y - z) - 2 u (x, y)}{| z |^{1 + 2 s}} d z . \end{aligned}

The above representation is useful to remove the singularity at the origin. Indeed, for any smooth function u, a second order Taylor expansion yields,

\begin{aligned} \frac{u (x, y + z) + u (x, y - z) - 2 u (x, y)}{| z |^{1 + 2 s}} ⩽ \frac{‖ D^{2} u ‖_{L^{\infty}}}{| z |^{2 s - 1}} \end{aligned}

which is integrable near 0 (for any fixed

s \in (0, 1)

). Therefore, since

u \in P (R^{2})

, one can get rid of the

P . V

. and write (2.17).

Remark 12.

One can observe that the operator ${(- Δ)}_{y}^{s}$ is indeed well-defined for any $u \in C^{2} (R^{2}) \cap L^{\infty} (R^{2})$ . If we take any non-zero constant, it does not belong to the Schwartz space $P (R^{2})$ but it is fractional harmonic.

For any $u \in P (R^{2})$ , the Fourier transform of u is well-defined and it is given by

\begin{aligned} F u (ξ) = \hat{u} (ξ) = \frac{1}{2 π} \int_{R^{2}} e^{- i ξ \cdot x} u (x) d x, for ξ = (ξ_{1}, ξ_{2}) \in R^{2} . \end{aligned}

Proposition 13.

For $s \in (0, 1)$ , let ${(- Δ)}_{y}^{s} : P (R^{2}) \to L^{2} (R^{2})$ be the fractional Laplacian operator given by (2.15). Then, for any $u \in P (R^{2})$ ,

\begin{aligned} {(- Δ)}_{y}^{s} u = F^{- 1} (| ξ_{2} |^{2 s} (F u)), \forall ξ = (ξ_{1}, ξ_{2}) \in R^{2} . \end{aligned}

(2.21)

Proof.

The Lemma 11 represents the fractional Laplacian w.r.t. y variable as a second order differential quotient given by (2.17). We denote by $K u$ the integral in (2.17), i.e.

\begin{aligned} K u (x, y) = - \frac{1}{2} C (s) \int_{R} \frac{u (x, y + z) + u (x, y - z) - 2 u (x, y)}{| z |^{1 + 2 s}} d z . \end{aligned}

K

is a linear operator and our goal is to look for a function

A : R \to R

such that

\begin{aligned} K u = F^{- 1} ((A (F u))) . \end{aligned}

(2.22)

Indeed, we seek for A satisfying

\begin{aligned} A (ξ_{2}) = | ξ_{2} |^{2 s} . \end{aligned}

(2.23)

Further, we have the following estimate

\begin{aligned} \frac{| u (x, y + z) + u (x, y - z) - 2 u (x, y) |}{| z |^{1 + 2 s}} & ⩽ 4 (χ_{B_{1}} | z |^{1 - 2 s} sup_{B_{1}} | D^{2} u | \\ + χ_{R ∖ B_{1}} | z |^{- 1 - 2 s} | u (x, y + z) + u (x, y - z) - 2 u (x, y) |) \in L^{1} (R^{2}) . \end{aligned}

Hence, by the Fubini–Tonelli theorem we can exchange the integral in z with the Fourier transform in

(x, y)

. Applying Fourier transform we obtain

\begin{aligned} A (ξ_{2}) (F u) (ξ) & = F (K u) \\ = - \frac{1}{2} C (s) \int_{R} \frac{F (u (x, y + z) + u (x, y - z) - 2 u (x, y))}{| z |^{1 + 2 s}} d z \\ = - \frac{1}{2} C (s) \int_{R} \frac{e^{i ξ_{2} z} + e^{- i ξ_{2} z} - 2}{| z |^{1 + 2 s}} d z (F u) (ξ) \\ = C (s) \int_{R} \frac{1 - \cos (ξ_{2} z)}{| z |^{1 + 2 s}} d z (F u) (ξ) . \end{aligned}

For claiming (2.23) it is enough to prove that

\begin{aligned} C (s) \int_{R} \frac{1 - \cos (ξ_{2} z)}{| z |^{1 + 2 s}} d z = | ξ_{2} |^{2 s} . \end{aligned}

(2.24)

Now, we consider a function

I : R \to R

defined by

\begin{aligned} I (ξ_{2}) = \int_{R} \frac{1 - \cos (ξ_{2} z)}{| z |^{1 + 2 s}} d z . \end{aligned}

We observe that

I (- ξ_{2}) = I (ξ_{2})

and therefore we have

\begin{aligned} I (ξ_{2}) = \int_{R} \frac{1 - \cos (ξ_{2} z)}{| z |^{1 + 2 s}} d z = | ξ_{2} |^{2 s} \int_{R} \frac{1 - \cos (t)}{| t |^{1 + 2 s}} d t = | ξ_{2} |^{2 s} C {(s)}^{- 1}, \end{aligned}

which proves (2.24) and hence the proof is complete.

Proposition 14.

Let $s \in (0, 1)$ . Then for any $u \in H^{s} (R^{2})$ we have

\begin{aligned} 2 C {(s)}^{- 1} \int_{R^{2}} | ξ_{2} |^{2 s} | F u (ξ) |^{2} d ξ = \int_{R^{3}} \frac{| u (x, y) - u (x, z) |^{2}}{| y - z |^{1 + 2 s}} d z d x d y, \end{aligned}

where

C (s)

is given by (2.16) and

H^{s} (R^{2})

is the fractional Sobolev space.

Proof.

First we compute

\begin{aligned} F (\frac{u (x, z + y) - u (x, y)}{| z |^{\frac{1}{2} + s}}) (ξ) & = \frac{1}{2 π} \frac{1}{| z |^{\frac{1}{2} + s}} \int_{R^{2}} e^{- i (ξ_{1}, ξ_{2}) \cdot (x, y)} (u (x, z + y) - u (x, y)) d x d y \\ = \frac{(e^{i ξ_{2} z} - 1)}{| z |^{\frac{1}{2} + s}} F u (ξ) . \end{aligned}

(2.25)

Now we calculate

\begin{aligned} \int_{R} {‖ F (\frac{u (x, z + y) - u (x, y)}{| z |^{\frac{1}{2} + s}}) ‖}_{L^{2} (R^{2})}^{2} d z & = \int_{R} \int_{R^{2}} \frac{| e^{i ξ_{2} z} - 1 |^{2}}{| z |^{1 + 2 s}} | F u (ξ_{1}, ξ_{2}) |^{2} d ξ_{1} d ξ_{2} d z \\ = 2 \int_{R} \int_{R^{2}} \frac{1 - \cos (ξ_{2} z)}{| z |^{1 + 2 s}} | F u (ξ) |^{2} d ξ d z \\ = 2 C {(s)}^{- 1} \int_{R^{2}} | ξ_{2} |^{2 s} | F u (ξ) |^{2} d ξ . \end{aligned}

(2.26)

On the other hand, by Plancherel formula we get

\begin{aligned} \int_{R} {‖ F (\frac{u (x, z + y) - u (x, y)}{| z |^{1 / 2 + s}}) ‖}_{L^{2} (R^{2})}^{2} d z & = \int_{R} {‖ \frac{u (x, z + y) - u (x, y)}{| z |^{1 / 2 + s}} ‖}_{L^{2} (R^{2})}^{2} d z \\ = \int_{R} (\int_{R^{2}} \frac{| u (x, z + y) - u (x, y) |^{2}}{| z |^{1 + 2 s}} d x d y) d z \\ = \int_{R^{2}} (\int_{R} \frac{| u (x, z + y) - u (x, y) |^{2}}{| z |^{1 + 2 s}} d z) d x d y \\ = \int_{R^{3}} \frac{| u (x, y) - u (x, z) |^{2}}{| y - z |^{1 + 2 s}} d z d x d y . \end{aligned}

(2.27)

From (2.26) and (2.27), we obtain

\begin{aligned} 2 C {(s)}^{- 1} \int_{R^{2}} | ξ_{2} |^{2 s} | F u (ξ) |^{2} d ξ = \int_{R^{3}} \frac{| u (x, y) - u (x, z) |^{2}}{| y - z |^{1 + 2 s}} d z d x d y . \end{aligned}

Remark 15.

Notice that the above Proposition is important in the sense that it gives us an equivalence relation between the Fourier norm and the Gagliardo semi-norm. Since our problem is of variational type, it is natural to work with the Gagliardo semi-norm than Fourier norm. Moreover, we also write (see [5, Remark 1.1])

\begin{aligned} {‖ {(- Δ)}_{y}^{s / 2} u ‖}_{2}^{2} = \frac{C (s)}{2} \int_{R^{3}} \frac{| u (x, y) - u (x, z) |^{2}}{| y - z |^{1 + 2 s}} d z d x d y . \end{aligned}

(2.28)

We define the fractional gradient w.r.t y variable for $u \in H^{s} (R^{2})$ as

\begin{aligned} {| D_{y}^{s} u (x, y) |}^{2} = \frac{C (s)}{2} \int_{R} \frac{| u (x, y) - u (x, z) |^{2}}{| y - z |^{1 + 2 s}} d z . \end{aligned}

(2.29)

The next two lemmas are due to [1, Lemma 2.2, Lemma 2.4]. The proof follows by similar arguments and therefore we omit the details. Lemma 16.

Let $v \in W^{1, \infty} (R^{2})$ be such that $supp (v) \subset B_{1} (0)$ . Then, there exists a constant $C > 0$ depending on s and $‖ v ‖_{1, \infty}$ such that

\begin{aligned} {| D_{y}^{s} v (x, y) |}^{2} ⩽ C min {1, {(x^{2} + y^{2})}^{- \frac{1 + 2 s}{2}}} . \end{aligned}

(2.30)

Lemma 17.

Let $s \in (0, 1)$ and $w \in L^{\infty} (R^{2})$ be such that there exist $α > 0$ and $C > 0$ such that

\begin{aligned} 0 ⩽ w (x, y) ⩽ C (x^{2} + y^{2})^{- \frac{α}{2}} . \end{aligned}

α > 2 s

, then

H^{1, s} (R^{2}) \subset\subset L^{2} (w d x d y; R^{2})

, where

L^{2} (w d x d y; R^{2})

is the weighted

L^{2}

-space with weight w and the symbol “

\subset\subset

” means the compact embedding.

Remark 18.

If $ϕ \in W^{1, \infty} (R^{2})$ has compact support, then $w = | D_{y}^{s} ϕ |^{2}$ verifies all the assumptions of the above lemma.

3. .Concentration–compactness

In this section, we recall the following result by Lions [8, Lemma 1.2], which we will be using further to prove the concentration-compactness results.

Lemma 19.
Let ν, μ be two non-negative, bounded measures on $R^{N}$ such that
$\begin{aligned} {(\int_{R^{N}} | ϕ |^{q} d ν)}^{\frac{1}{q}} ⩽ C {(\int_{R^{N}} | ϕ |^{p} d μ)}^{\frac{1}{p}}, \forall ϕ \in C_{c}^{\infty} (R^{N}), \end{aligned}$
for some constant $C > 0$ and $1 ⩽ p < q < \infty$ . Then there exist a countable set ${x_{k} \in R^{N} : k \in K}$ and $ν_{k} \in (0, \infty)$ such that $ν = \sum_{k \in K} ν_{k} δ_{x_{k}} .$
Lemma 20 ([2], Brezis–Lieb Theorem 1)

Let $r ⩾ 1$ and ${u_{n}}_{n \in N} \subset L^{r} (R^{N})$ be a bounded sequence such that $u_{n} \to u$ a.e. in $R^{N}$ , then

\begin{aligned} ‖ u_{n} ‖_{L^{r} (R^{N})}^{r} - ‖ u_{n} - u ‖_{L^{r} (R^{N})}^{r} = ‖ u ‖_{L^{r} (R^{N})}^{r} + o_{n} (1) . \end{aligned}

Next we recall the following lemma which establishes the embedding between $H^{1, s} (R^{2})$ and $L^{q} (R^{2})$ for some q.

Lemma 21 ([4, Theorem 1.1])

Let $s \in (0, 1)$ and $2 ⩽ q ⩽ 2_{s} = \frac{2 (1 + s)}{1 - s}$ . For any $u \in H^{1, s} (R^{2})$ , there exists a constant $C_{q, s} > 0$ such that

\begin{aligned} \int_{R^{2}} | u |^{q} d x d y ⩽ C_{q, s} {(\int_{R^{2}} | u |^{2} d x d y)}^{\frac{q}{2} - \frac{(q - 2) (s + 1)}{4 s}} {(\int_{R^{2}} | \partial_{x} u |^{2} d x d y)}^{\frac{q - 2}{4}} \times {(\int_{R^{2}} {| {(- Δ)}_{y}^{s / 2} u |}^{2} d x d y)}^{\frac{q - 2}{4 s}} . \end{aligned}

(3.1)

Note that all the exponents on the right hand side of (3.1) are positive for $2 < q < 2_{s}$ . Raising (3.1) to power $\frac{2}{q}$ , we obtain that

\begin{aligned} {(\int_{R^{2}} | u |^{q} d x d y)}^{2 / q} ⩽ C_{q, s} {(\int_{R^{2}} | u |^{2} d x d y)}^{1 - \frac{2 (q - 2) (s + 1)}{4 s q}} {(\int_{R^{2}} | \partial_{x} u |^{2} d x d y)}^{\frac{2 (q - 2)}{4 q}} \times {(\int_{R^{2}} {| {(- Δ)}_{y}^{s / 2} u |}^{2} d x d y)}^{\frac{2 (q - 2)}{4 s q}} . \end{aligned}

As a consequence, for all

q : 2 < q < 2_{s}

, we can apply the Young’s inequality to obtain the following

\begin{aligned} {(\int_{R^{2}} | u |^{q} d x d y)}^{2 / q} ⩽ C \int_{R^{2}} (| u |^{2} + | \partial_{x} u |^{2} + {| {(- Δ)}_{y}^{s / 2} u |}^{2}) d x d y . \end{aligned}

(3.2)

Note that (3.2) is equivalent to the Sobolev type embedding

H^{1, s} (R^{2}) ↪ L^{q} (R^{2})

, which holds under the assumption

s > 0

and

2 < q ⩽ 2_{s}

(see also [4, Remark 2.2]). Let us define

\begin{aligned} Λ_{i} := inf_{u \in H^{1, s_{i}} (R^{2}) ∖ {0}} \frac{\int_{R^{2}} (| u |^{2} + | \partial_{x} u |^{2} + | {(- Δ)}_{y}^{s_{i} / 2} u |^{2}) d x d y}{{(\int_{R^{2}} | u |^{2_{s_{i}}} d x d y)}^{2 / 2_{s_{i}}}}, i = 1, 2. \end{aligned}

(3.3)

By definition it follows that

\begin{aligned} Λ_{i} ‖ u ‖_{2_{s_{i}}}^{2} ⩽ ‖ u ‖_{H^{1, s_{i}} (R^{2})}^{2}, i = 1, 2, \end{aligned}

(3.4)

where

Λ_{i} > 0 (i = 1, 2)

by embedding (3.2). For the sake of simplicity, let us first deal with the case

s_{i} = s

and

Λ_{i} = Λ

for

i = 1, 2

. It is given that for

u \in H^{1, s} (R^{2})

\begin{aligned} {(- Δ)}_{y}^{s / 2} u (x, y) = C (s) P . V . \int_{R} \frac{u (x, y) - u (x, z)}{| y - z |^{1 + s}} d z . \end{aligned}

Observe that

‖ D_{y}^{s} u ‖_{L^{2} (R^{2})}^{2} = ‖ {(- Δ)}_{y}^{s / 2} u ‖_{L^{2} (R^{2})}^{2} = \frac{C (s)}{2} \int_{R^{3}} \frac{| u (x, y) - u (x, z) |^{2}}{| y - z |^{1 + 2 s}} d z d x d y

. Let

ϕ \in C_{c}^{\infty} (R^{2})

, then by critical Sobolev type embedding (3.4), we obtain

\begin{aligned} Λ^{1 / 2} ‖ ϕ u_{k} ‖_{2_{s}} ⩽ ‖ ϕ u_{k} ‖_{H^{1, s} (R^{2})}, \end{aligned}

(3.5)

where the sequence

{u_{k}}

converges weakly to u in

H^{1, s} (R^{2})

. First we assume that

u = 0

, then

\begin{aligned} ‖ ϕ u_{k} ‖_{2_{s}} = {(\int_{R^{2}} | ϕ u_{k} |^{2_{s}} d x d y)}^{\frac{1}{2_{s}}} = {(\int_{R^{2}} | ϕ |^{2_{s}} | u_{k} |^{2_{s}} d x d y)}^{\frac{1}{2_{s}}} . \end{aligned}

Let

ν_{k} := | u_{k} |^{2_{s}} d x d y \overset{*}{⇀} ν

and

μ_{k} := (| u_{k} |^{2} + | \partial_{x} u_{k} |^{2} + | D_{y}^{s} u_{k} |^{2}) d x d y \overset{*}{⇀} μ

in the sense of measure. Then

\begin{aligned} ‖ ϕ u_{k} ‖_{2_{s}} = {(\int_{R^{2}} | ϕ |^{2_{s}} | u_{k} |^{2_{s}} d x d y)}^{\frac{1}{2_{s}}} \to {(\int_{R^{2}} | ϕ |^{2_{s}} d ν)}^{\frac{1}{2_{s}}} as k \to \infty . \end{aligned}

(3.6)

Now

\begin{aligned} ‖ ϕ u_{k} ‖_{H^{1, s} (R^{2})}^{2} & = \int_{R^{2}} (| ϕ u_{k} |^{2} + {| \partial_{x} (ϕ u_{k}) |}^{2} + {| D_{y}^{s} (ϕ u_{k}) |}^{2}) d x d y \\ = \underset{I}{\underset{⏟}{\int_{R^{2}} | ϕ u_{k} |^{2} d x d y}} + \underset{II}{\underset{⏟}{\int_{R^{2}} {| \partial_{x} (ϕ u_{k}) |}^{2} d x d y}} + \underset{III}{\underset{⏟}{\int_{R^{2}} {| D_{y}^{s} (ϕ u_{k}) |}^{2} d x d y}} . \end{aligned}

\begin{aligned} III & = \int_{R^{2}} {| D_{y}^{s} (ϕ u_{k}) |}^{2} d x d y = \int_{R^{2}} {| {(- Δ)}_{y}^{s / 2} (ϕ u_{k}) |}^{2} d x d y \\ = \frac{C (s)}{2} \int_{R^{3}} \frac{| (ϕ u_{k}) (x, y) - (ϕ u_{k}) (x, z) |^{2}}{| y - z |^{1 + 2 s}} d z d x d y \\ = \frac{C (s)}{2} \int_{R^{3}} \frac{| ϕ (x, y + h) u_{k} (x, y + h) - ϕ (x, y) u_{k} (x, y) |^{2}}{| h |^{1 + 2 s}} d h d x d y \\ = \frac{C (s)}{2} \int_{R^{3}} ((| ϕ (x, y + h) u_{k} (x, y + h) - ϕ (x, y) u_{k} (x, y + h) + ϕ (x, y) u_{k} (x, y + h) - ϕ (x, y) u_{k} (x, y) |^{2}) / (| h |^{1 + 2 s})) d h d x d y \\ = \frac{C (s)}{2} \int_{R^{3}} \frac{| (ϕ (x, y + h) - ϕ (x, y)) u_{k} (x, y + h) + ϕ (x, y) (u_{k} (x, y + h) - u_{k} (x, y)) |^{2}}{| h |^{1 + 2 s}} d h \\ \times d x d y . \end{aligned}

Now the application of Minkowski’s inequality gives the following estimate

\begin{aligned} III & ⩽ \frac{C (s)}{2} {[\int_{R^{3}} (\frac{| ϕ (x, y + h) - ϕ (x, y) |^{2} | u_{k} (x, y + h) |^{2}}{| h |^{1 + 2 s}} d h d x d y)}^{1 / 2} \\ + {{(\int_{R^{3}} \frac{| u_{k} (x, y + h) - u_{k} (x, y) |^{2} | ϕ (x, y) |^{2}}{| h |^{1 + 2 s}} d h d x d y)}^{1 / 2}]}^{2} \\ = {[(\frac{C (s)}{2} \int_{R^{3}} (\frac{| ϕ (x, y + h) - ϕ (x, y) |^{2} | u_{k} (x, y + h) |^{2}}{| h |^{1 + 2 s}} d h d x d y)}^{1 / 2} \\ + {{(\frac{C (s)}{2} \int_{R^{3}} \frac{| u_{k} (x, y + h) - u_{k} (x, y) |^{2} | ϕ (x, y) |^{2}}{| h |^{1 + 2 s}} d h d x d y)}^{1 / 2}]}^{2} \\ = [(\int_{R^{2}} {({| ϕ (x, y) |}^{2} {| D_{y}^{s} u_{k} (x, y) |}^{2} d x d y)}^{1 / 2} \\ + {{\underset{IV}{\underset{⏟}{(\frac{C (s)}{2} \int_{R^{3}} \frac{| ϕ (x, y + h) - ϕ (x, y) |^{2} | u_{k} (x, y + h) |^{2}}{| h |^{1 + 2 s}} d h d x d y)}}}^{1 / 2}]}^{2} . \end{aligned}

(3.7)

\begin{aligned} IV & = \frac{C (s)}{2} \int_{R^{3}} \frac{| ϕ (x, y + h) - ϕ (x, y) |^{2} | u_{k} (x, y + h) |^{2}}{| h |^{1 + 2 s}} d h d x d y \\ = \frac{C (s)}{2} \int_{R^{3}} \frac{| ϕ (x, t) - ϕ (x, t - h) |^{2} | u_{k} (x, t) |^{2}}{| h |^{1 + 2 s}} d h d x d t \\ = \frac{C (s)}{2} \int_{R^{3}} \frac{| ϕ (x, t) - ϕ (x, t + \hat{h}) |^{2} | u_{k} (x, t) |^{2}}{| \hat{h} |^{1 + 2 s}} d \hat{h} d x d t \\ = \frac{C (s)}{2} \int_{R^{3}} \frac{| ϕ (x, y) - ϕ (x, y + h) |^{2} | u_{k} (x, y) |^{2}}{| h |^{1 + 2 s}} d h d x d y \\ = \int_{R^{2}} {| u_{k} (x, y) |}^{2} {| D_{y}^{s} ϕ (x, y) |}^{2} d x d y . \end{aligned}

(3.8)

From (3.7) and (3.8), we deduce that

\begin{aligned} {III}^{1 / 2} ⩽ {(\int_{R^{2}} {| ϕ (x, y) |}^{2} {| D_{y}^{s} u_{k} (x, y) |}^{2} d x d y)}^{1 / 2} + {(\int_{R^{2}} {| u_{k} (x, y) |}^{2} {| D_{y}^{s} ϕ (x, y) |}^{2} d x d y)}^{1 / 2} . \end{aligned}

(3.9)

Using Remark 18, the weight

w (x, y) := | D_{y}^{s} ϕ (x, y) |^{2}

verifies all the hypothesis of Lemma 17. Therefore, the sequence

{u_{k}}

converges strongly to

u = 0

L^{2} (w d x d y; R^{2})

. Hence, we obtain

\begin{aligned} lim_{k \to \infty} {III}^{1 / 2} ⩽ lim_{k \to \infty} {(\int_{R^{2}} {| ϕ (x, y) |}^{2} {| D_{y}^{s} u_{k} (x, y) |}^{2} d x d y)}^{1 / 2} . \end{aligned}

(3.10)

Further,

\begin{aligned} I = \int_{R^{2}} | ϕ |^{2} | u_{k} |^{2} d x d y \to \int_{R^{2}} | ϕ |^{2} | u |^{2} d x d y as k \to \infty \end{aligned}

(3.11)

and

\begin{aligned} {II}^{1 / 2} & = {(\int_{R^{2}} {| \partial_{x} (ϕ u_{k}) |}^{2} d x d y)}^{1 / 2} = {(\int_{R^{2}} {| ϕ \partial_{x} (u_{k}) + u_{k} \partial_{x} (ϕ) |}^{2} d x d y)}^{1 / 2} = {‖ ϕ \partial_{x} (u_{k}) + u_{k} \partial_{x} (ϕ) ‖}_{2} \\ ⩽ {‖ ϕ \partial_{x} (u_{k}) ‖}_{2} + {‖ u_{k} \partial_{x} (ϕ) ‖}_{2} (by Minkowski's inequality) . \end{aligned}

The second term in the last inequality tends to 0 as

k \to \infty

using a similar set of arguments as in Lemma 17. Subsequently, we have

\begin{aligned} lim_{k \to \infty} {II}^{1 / 2} ⩽ lim_{k \to \infty} {‖ ϕ \partial_{x} (u_{k}) ‖}_{2} . \end{aligned}

(3.12)

Thus we have

\begin{aligned} Λ lim_{k \to \infty} ‖ ϕ u_{k} ‖_{2_{s}}^{2} ⩽ lim_{k \to \infty} (\int_{R^{2}} | ϕ |^{2} (| u_{k} |^{2} + | \partial_{x} u_{k} |^{2} + | D_{y}^{s} u_{k} |^{2}) d x d y) . \end{aligned}

Further,

\begin{aligned} Λ lim_{k \to \infty} {(\int_{R^{2}} | ϕ |^{2_{s}} | u_{k} |^{2_{s}} d x d y)}^{2 / 2_{s}} & ⩽ lim_{k \to \infty} (\int_{R^{2}} | ϕ |^{2} (| u_{k} |^{2} + | \partial_{x} u_{k} |^{2} + {| D_{y}^{s} u_{k} |}^{2}) d x d y) \\ Λ lim_{k \to \infty} {(\int_{R^{2}} | ϕ |^{2_{s}} d ν_{k})}^{2 / 2_{s}} & ⩽ lim_{k \to \infty} (\int_{R^{2}} | ϕ |^{2} d μ_{k}) \\ Λ {(\int_{R^{2}} | ϕ |^{2_{s}} d ν)}^{2 / 2_{s}} & ⩽ (\int_{R^{2}} | ϕ |^{2} d μ) . \end{aligned}

Now by Lemma 19, there exists a countable set

J

, points

{(x_{j}, y_{j})}_{j \in J} \subset R^{2}

and positive weights

{ν_{j}}, {μ_{j}} \subset R

such that

\begin{aligned} ν = \sum_{j \in J} ν_{j} δ_{(x_{j}, y_{j})} + ν_{0} δ_{(0, 0)}, and μ ⩾ \sum_{j \in J} μ_{j} δ_{(x_{j}, y_{j})} + μ_{0} δ_{(0, 0)}, \end{aligned}

where

δ_{(x_{j}, y_{j})}

and

δ_{(0, 0)}

are Dirac measures centred at points

(x_{j}, y_{j})

and

(0, 0)

respectively.

Now if $u \neq 0$ , we replace $u_{k}$ by $v_{k} = u_{k} - u$ . Since $u_{k} ⇀ u$ weakly in $H^{1, s} (R^{2})$ , the sequence $v_{k} ⇀ 0$ in $H^{1, s} (R^{2})$ . Let $ϕ \in C_{b}^{0} (R^{2})$ (the set of all continuous bounded functions), then Brezis–Lieb Lemma 20 yields

\begin{aligned} \int_{R^{2}} | u_{n} |^{2_{s}} ϕ d x d y = \int_{R^{2}} | u |^{2_{s}} ϕ d x d y + \int_{R^{2}} | u_{n} - u |^{2_{s}} ϕ d x d y + o_{n} (1) . \end{aligned}

Consequently we obtain

\begin{aligned} | u_{n} |^{2_{s}} \overset{*}{⇀} | u |^{2_{s}} + ν = | u |^{2_{s}} + \sum_{j \in J} ν_{j} δ_{(x_{j}, y_{j})} + ν_{0} δ_{(0, 0)} . \end{aligned}

(3.13)

Applying the Brezis–Lieb lemma again, we obtain

\begin{aligned} \int_{R^{2}} (| u_{n} |^{2} + {| \partial_{x} (u_{n}) |}^{2} + {| D_{y}^{s} u_{n} |}^{2}) ϕ d x d y - \int_{R^{2}} (| u_{n} - u |^{2} + {| \partial_{x} (u_{n} - u) |}^{2} + {| D_{y}^{s} (u_{n} - u) |}^{2}) ϕ d x d y \\ = \int_{R^{2}} (| u |^{2} + {| \partial_{x} (u) |}^{2} + {| D_{y}^{s} u |}^{2}) ϕ d x d y + o_{n} (1), \end{aligned}

which further gives

\begin{aligned} | u_{n} |^{2} + {| \partial_{x} (u_{n}) |}^{2} + {| D_{y}^{s} u_{n} |}^{2} \overset{*}{⇀} | u |^{2} + {| \partial_{x} (u) |}^{2} + {| D_{y}^{s} u |}^{2} + μ ⩾ | u |^{2} + {| \partial_{x} (u) |}^{2} + {| D_{y}^{s} u |}^{2} + \sum_{j \in J} μ_{j} δ_{(x_{j}, y_{j})} + μ_{0} δ_{(0, 0)} . \end{aligned}

(3.14)

Moreover, we also have

\begin{aligned} μ_{j} ⩾ Λ ν_{j}^{2 / 2_{s}}, \forall j \in J . \end{aligned}

(3.15)

Lemma 22.

Suppose $α + β < min {2_{s_{1}}, 2_{s_{2}}}$ or $α + β = min {2_{s_{1}}, 2_{s_{2}}} < max {2_{s_{1}}, 2_{s_{2}}}$ and h satisfies (H). Then the functional $J_{κ}$ satisfies the (PS)-condition at level c satisfying

\begin{aligned} c < min {\frac{s_{1}}{1 + s_{1}} Λ_{1}^{\frac{1 + s_{1}}{2 s_{1}}}, \frac{s_{2}}{1 + s_{2}} Λ_{2}^{\frac{1 + s_{2}}{2 s_{2}}}} . \end{aligned}

(3.16)

Proof.

Let ${(u_{n}, v_{n})}$ be a (PS)-sequence for $J_{κ}$ in $D$ . Then ${(u_{n}, v_{n})}$ is a bounded sequence in $D$ (see Lemma 10) and by reflexivity there exists a subsequence still denoted by ${(u_{n}, v_{n})}$ itself and $(u, v) \in D$ satisfying the following

\begin{aligned} \begin{aligned} (u_{n}, v_{n}) ⇀ (u, v) & weakly in D, \\ (u_{n}, v_{n}) \to (u, v) & strongly in L_{loc}^{q_{1}} (R^{2}) \times L_{loc}^{q_{2}} (R^{2}) for 1 ⩽ q_{1} < 2_{s_{1}}, 1 ⩽ q_{2} < 2_{s_{2}}, \\ (u_{n}, v_{n}) \to (u, v) & a . e . in R^{2} . \end{aligned} \end{aligned}

Now using (3.13), (3.14) and (3.15), which are analogous to the concentration–compactness principle of Bonder [1, Theorem 1.1], there exist a subsequence, still denoted as

{(u_{n}, v_{n})}

, two at most countable sets of points

{(x_{j}, y_{j})}_{j \in J} \subset R^{2}

and

{({\bar{x}}_{k}, {\bar{y}}_{k})}_{k \in K} \subset R^{2}

, and non-negative numbers

\begin{aligned} {(μ_{j}, ν_{j})}_{j \in J}, {({\bar{μ}}_{k}, {\bar{ν}}_{k})}_{k \in K}, μ_{0}, ν_{0}, {\bar{μ}}_{0} and {\bar{ν}}_{0} \end{aligned}

such that the following convergences hold weakly^∗ in the sense of measures,

\begin{aligned} | u_{n} |^{2} + | \partial_{x} u_{n} |^{2} + {| D_{y}^{s_{1}} u_{n} |}^{2} ⇀ d μ ⩾ | u |^{2} + | \partial_{x} u |^{2} + {| D_{y}^{s_{1}} u |}^{2} + \sum_{j \in J} μ_{j} δ_{(x_{j}, y_{j})} + μ_{0} δ_{(0, 0)}, \\ | v_{n} |^{2} + | \partial_{x} v_{n} |^{2} + {| D_{y}^{s_{2}} v_{n} |}^{2} ⇀ d \bar{μ} ⩾ | v |^{2} + | \partial_{x} v |^{2} + {| D_{y}^{s_{2}} v |}^{2} + \sum_{k \in K} {\bar{μ}}_{k} δ_{({\bar{x}}_{k}, {\bar{y}}_{k})} + μ_{0} δ_{(0, 0)}, \\ | u_{n} |^{2_{s_{1}}} ⇀ d ν = | u |^{2_{s_{1}}} + \sum_{j \in J} ν_{j} δ_{(x_{j}, y_{j})} + ν_{0} δ_{(0, 0)}, \\ | v_{n} |^{2_{s_{2}}} ⇀ d \bar{ν} = | v |^{2_{s_{2}}} + \sum_{k \in K} {\bar{ν}}_{k} δ_{({\bar{x}}_{k}, {\bar{y}}_{k})} + {\bar{ν}}_{0} δ_{(0, 0)} . \end{aligned}

(3.17)

Moreover,

\begin{aligned} Λ_{1} ν_{j}^{2 / 2_{s_{1}}} & ⩽ μ_{j} for all j \in J \cup {0}, \\ Λ_{2} {\bar{ν}}_{k}^{2 / 2_{s_{2}}} & ⩽ {\bar{μ}}_{k} for all k \in K \cup {0}, \end{aligned}

(3.18)

where

δ_{(0, 0)}

δ_{(x_{j}, y_{j})}

δ_{({\bar{x}}_{k}, {\bar{y}}_{k})}

are the Dirac measures at the points

(0, 0)

(x_{j}, y_{j}) and ({\bar{x}}_{k}, {\bar{y}}_{k})

R^{2}

respectively. We denote the concentration of the sequence

{u_{n}}

at infinity by the following numbers

\begin{aligned} ν_{\infty} & = lim_{R \to \infty} \underset{n \to \infty}{lim sup} \int_{\sqrt{x^{2} + y^{2}} > R} | u_{n} |^{2_{s_{1}}} d x d y, \\ μ_{\infty} & = lim_{R \to \infty} \underset{n \to \infty}{lim sup} \int_{\sqrt{x^{2} + y^{2}} > R} (| u_{n} |^{2} + | \partial_{x} u_{n} |^{2} + {| D_{y}^{s_{1}} u_{n} |}^{2}) d x d y . \end{aligned}

(3.19)

We similarly define the concentrations of the sequence

{v_{n}}

at infinity by the numbers

{\bar{ν}}_{\infty}

and

{\bar{μ}}_{\infty}

. Now let us take a smooth cut-off function centered at

{(x_{j}, y_{j})}_{j \in J} \subset R^{2}

and defined as

\begin{aligned} Ψ_{j, ϵ} = 1 in B_{\frac{ϵ}{2}} (x_{j}, y_{j}), Ψ_{j, ϵ} = 0 in B_{ϵ}^{c} (x_{j}, y_{j}), 0 ⩽ Ψ_{j, ϵ} ⩽ 1 and | \nabla Ψ_{j, ϵ} | ⩽ \frac{4}{ϵ}, \end{aligned}

(3.20)

where

B_{r} (x_{j}, y_{j}) = {(x, y) \in R^{2} : (x - x_{j})^{2} + (y - y_{j})^{2} < r^{2}}

. Testing

J_{κ}^{^{'}} (u_{n}, v_{n})

with

(u_{n} Ψ_{j, ϵ}, 0)

we have

\begin{aligned} ⟨ J_{κ}^{^{'}} (u_{n}, v_{n}) | (u_{n} Ψ_{j, ϵ}, 0) ⟩ & = \int_{R^{2}} (\partial_{x} u_{n} \partial_{x} (u_{n} Ψ_{j, ϵ}) + D_{y}^{s_{1}} u_{n} D_{y}^{s_{1}} (u_{n} Ψ_{j, ϵ}) + u_{n} u_{n} Ψ_{j, ϵ} - | u_{n} |^{2_{s_{1}} - 2} u_{n} u_{n} Ψ_{j, ϵ}) d x d y \\ - κ α \int_{R^{2}} h (x, y) | u_{n} |^{α - 2} u_{n} u_{n} Ψ_{j, ϵ} | v_{n} |^{β} d x d y \\ = \int_{R^{2}} (\partial_{x} u_{n} (u_{n} \partial_{x} (Ψ_{j, ϵ}) + Ψ_{j, ϵ} \partial_{x} u_{n}) + D_{y}^{s_{1}} u_{n} D_{y}^{s_{1}} (u_{n} Ψ_{j, ϵ}) + u_{n}^{2} Ψ_{j, ϵ} - | u_{n} |^{2_{s_{1}}} Ψ_{j, ϵ}) d x d y \\ - κ α \int_{R^{2}} h (x, y) | u_{n} |^{α} Ψ_{j, ϵ} | v_{n} |^{β} d x d y . \end{aligned}

Now

\begin{aligned} \int_{R^{2}} D_{y}^{s_{1}} u_{n} D_{y}^{s_{1}} (u_{n} Ψ_{j, ϵ}) d x d y & = \frac{C (s_{1})}{2} \int_{R^{3}} \frac{| u_{n} (x, y) - u_{n} (x, z) |^{2}}{| y - z |^{1 + 2 s_{1}}} Ψ_{j, ϵ} (x, y) d z d x d y \\ + \underset{I}{\underset{⏟}{\frac{C (s_{1})}{2} \int_{R^{3}} \frac{(u_{n} (x, y) - u_{n} (x, z)) (Ψ_{j, ϵ} (x, y) - Ψ_{j, ϵ} (x, z))}{| y - z |^{1 + 2 s_{1}}} u_{n} (x, z) d z d x d y}} . \\ I & = \frac{C (s_{1})}{2} \int_{R^{3}} \frac{(u_{n} (x, y) - u_{n} (x, z)) (Ψ_{j, ϵ} (x, y)) - Ψ_{j, ϵ} (x, z)}{| y - z |^{1 + 2 s_{1}}} u_{n} (x, z) d z d x d y . \end{aligned}

(3.21)

Since

{u_{n} Ψ_{j, ϵ}}

is a bounded sequence in

H^{1, s} (R^{2})

, the Hölder’s inequality gives

\begin{aligned} I & ⩽ \frac{C (s_{1})}{2} {(\int_{R^{3}} \frac{| u_{n} (x, y) - u_{n} (x, z) |^{2}}{| y - z |^{1 + 2 s_{1}}} d z d x d y)}^{1 / 2} \times {(\int_{R^{3}} \frac{| Ψ_{j, ϵ} (x, y) - Ψ_{j, ϵ} (x, z) |^{2}}{| y - z |^{1 + 2 s_{1}}} | u_{n} (x, z) |^{2} d z d x d y)}^{1 / 2} \\ ⩽ \frac{C (s_{1})}{2} A (s) \int_{R^{3}} \frac{| Ψ_{j, ϵ} (x, y) - Ψ_{j, ϵ} (x, z) |^{2}}{| y - z |^{1 + 2 s_{1}}} {| u_{n} (x, z) |}^{2} d z d x d y = A (s) \int_{R^{2}} {| u_{n} (x, z) |}^{2} {| D_{y}^{s_{1}} Ψ_{j, ϵ} |}^{2} d x d z . \end{aligned}

Using Lemma 2.3 of Xiang et al. [9], we obtain

\begin{aligned} lim_{ϵ \to 0} lim_{n \to \infty} I = 0. \end{aligned}

(3.22)

Now

\begin{aligned} \int_{R^{2}} (\partial_{x} u_{n} (u_{n} \partial_{x} Ψ_{j, ϵ} + Ψ_{j, ϵ} \partial_{x} u_{n}) d x d y = \int_{R^{2}} \partial_{x} u_{n} \partial_{x} Ψ_{j, ϵ} u_{n} d x d y + \int_{R^{2}} | \partial_{x} u_{n} |^{2} Ψ_{j, ϵ} d x d y . \end{aligned}

Note that

\begin{aligned} lim_{ϵ \to 0} \underset{n \to \infty}{lim sup} \int_{R^{2}} \partial_{x} u_{n} \partial_{x} (Ψ_{j, ϵ}) u_{n} d x d y = 0. \end{aligned}

Therefore we have

\begin{aligned} lim_{ϵ \to 0} \underset{n \to \infty}{lim sup} \int_{R^{2}} (\partial_{x} u_{n} (u_{n} \partial_{x} Ψ_{j, ϵ} + Ψ_{j, ϵ} \partial_{x} u_{n}) d x d y = lim_{ϵ \to 0} \underset{n \to \infty}{lim sup} \int_{R^{2}} | \partial_{x} u_{n} |^{2} Ψ_{j, ϵ} d x d y . \end{aligned}

Thus we obtain

\begin{aligned} 0 & = lim_{n \to \infty} ⟨ J_{κ}^{^{'}} (u_{n}, v_{n}) | (u_{n} Ψ_{j, ϵ}, 0) ⟩ = lim_{n \to \infty} (\frac{C (s_{1})}{2} \int_{R^{3}} \frac{| u_{n} (x, y) - u_{n} (x, z) |^{2}}{| y - z |^{1 + 2 s_{1}}} Ψ_{j, ϵ} (x, y) d z d x d y \\ + \frac{C (s_{1})}{2} \int_{R^{3}} \frac{(u_{n} (x, y) - u_{n} (x, z)) \cdot (Ψ_{j, ϵ} (x, y) - Ψ_{j, ϵ} (x, z)}{| y - z |^{1 + 2 s_{1}}} u_{n} (x, z) d z d x d y \\ + \int_{R^{2}} u_{n}^{2} Ψ_{j, ϵ} + \int_{R^{2}} | \partial_{x} u_{n} |^{2} Ψ_{j, ϵ} d x d y + \int_{R^{2}} \partial_{x} u_{n} \partial (Ψ_{j, ϵ}) u_{n} d x d y \\ - \int_{R^{2}} | u_{n} |^{2_{s_{1}}} Ψ_{j, ϵ} d x d y - κ α \int_{R^{2}} h (x, y) | u_{n} |^{α} | v_{n} |^{β} Ψ_{j, ϵ} d x d y) . \end{aligned}

Taking the limit

ϵ \to 0

in the above equality, we obtain

\begin{aligned} 0 = lim_{ϵ \to 0^{+}} \int_{R^{2}} Ψ_{j, ϵ} d μ - lim_{ϵ \to 0^{+}} \int_{R^{2}} Ψ_{j, ϵ} d ν - lim_{ϵ \to 0^{+}} lim_{n \to \infty} κ α \int_{R^{2}} h (x, y) | u_{n} |^{α} | v_{n} |^{β} Ψ_{j, ϵ} d x d y . \end{aligned}

(3.23)

Notice that

0 \notin supp (Ψ_{j, ϵ})

for

ϵ

being sufficiently small and also

h \in L^{1} (R^{2}) \cap L^{\infty} (R^{2})

, we deduce that

\begin{aligned} \int_{R^{2}} Ψ_{j, ϵ} d μ & ⩾ \int_{R^{2}} Ψ_{j, ϵ} (| u |^{2} + | \partial_{x} u |^{2} + {| {(- Δ)}_{y}^{s_{1} / 2} u |}^{2}) d x d y + \sum_{k \in J} μ_{k} δ_{(x_{k}, y_{k})} Ψ_{j, ϵ} + μ_{0} δ_{0} (Ψ_{j, ϵ}) \\ = \int_{R^{2}} Ψ_{j, ϵ} (| u |^{2} + | \partial_{x} u |^{2} + {| {(- Δ)}_{y}^{s_{1} / 2} u |}^{2}) d x d y + \sum_{k \in J} μ_{k} Ψ_{j, ϵ} (x_{k}, y_{k}) + μ_{0} Ψ_{j, ϵ} (0, 0) . \end{aligned}

Consequently we have

\begin{aligned} lim_{ϵ \to 0^{+}} \int_{R^{2}} Ψ_{j, ϵ} d μ ⩾ μ_{j}, \end{aligned}

(3.24)

and similarly we obtain

\begin{aligned} lim_{ϵ \to 0^{+}} \int_{R^{2}} Ψ_{j, ϵ} d ν = ν_{j} . \end{aligned}

(3.25)

Observe that both conditions

α + β < min {2_{s_{1}}, 2_{s_{2}}}

and

α + β = min {2_{s_{1}}, 2_{s_{2}}} < max {2_{s_{1}}, 2_{s_{2}}}

imply

\frac{α}{2_{s_{1}}} + \frac{β}{2_{s_{2}}} < 1

. Now using Hölder’s inequality, we estimate the following integral

\begin{aligned} \int_{R^{2}} h (x, y) | u_{n} |^{α} | v_{n} |^{β} Ψ_{j, ϵ} d x d y & = \int_{R^{2}} (h (x, y) Ψ_{j, ϵ} (x, y))^{1 - \frac{α}{2_{s_{1}}} - \frac{β}{2_{s_{2}}}} (h (x, y) Ψ_{j, ϵ} (x, y))^{\frac{α}{2_{s_{1}}} + \frac{β}{2_{s_{2}}}} | u_{n} |^{α} | v_{n} |^{β} d x d y \\ ⩽ {(\int_{R^{2}} h (x, y) Ψ_{j, ϵ} (x, y) d x d y)}^{1 - \frac{α}{2_{s_{1}}} - \frac{β}{2_{s_{2}}}} {(\int_{R^{2}} h (x, y) | u_{n} |^{2_{s_{1}}} Ψ_{j, ϵ} (x, y) d x d y)}^{\frac{α}{2_{s_{1}}}} \\ \times {(\int_{R^{2}} h (x, y) | v_{n} |^{2_{s_{2}}} Ψ_{j, ϵ} (x, y) d x d y)}^{\frac{β}{2_{s_{2}}}} . \end{aligned}

The first integral on the RHS of the above inequality tends to 0 as

ϵ \to 0^{+}

and the rest two integrals are bounded as limit

n \to \infty

. Hence, we can conclude that

\begin{aligned} lim_{ϵ \to 0^{+}} lim_{n \to \infty} \int_{R^{2}} h (x, y) | u_{n} |^{α} | v_{n} |^{β} Ψ_{j, ϵ} d x d y = 0. \end{aligned}

(3.26)

Now combining (3.23), (3.24), (3.25) and (3.26) gives

\begin{aligned} μ_{j} ⩽ ν_{j} as ϵ \to 0. \end{aligned}

(3.27)

From (3.18) and (3.27), we have the following

\begin{aligned} Λ_{1} ν_{j}^{2 / 2_{s_{1}}} ⩽ μ_{j} ⩽ ν_{j} \end{aligned}

which further gives

\begin{aligned} either ν_{j} = 0 or ν_{j} ⩾ Λ_{1}^{\frac{1 + s_{1}}{2 s_{1}}} for all j \in J and, J is finite . \end{aligned}

(3.28)

Also, we obtain

\begin{aligned} either {\bar{ν}}_{k} = 0 or {\bar{ν}}_{k} ⩾ Λ_{2}^{\frac{1 + s_{2}}{2 s_{2}}} for all k \in K and K is finite . \end{aligned}

(3.29)

It is already known that

\begin{aligned} c & = (\frac{1}{2} - \frac{1}{α + β}) {‖ (u_{n}, v_{n}) ‖}_{D}^{2} + (\frac{1}{α + β} - \frac{1}{2_{s_{1}}}) ‖ u_{n} ‖_{2_{s_{1}}}^{2_{s_{1}}} + (\frac{1}{α + β} - \frac{1}{2_{s_{2}}}) ‖ v_{n} ‖_{2_{s_{2}}}^{2_{s_{2}}} + o_{n} (1) \\ ⩾ (\frac{1}{2} - \frac{1}{α + β}) [{‖ (u, v) ‖}_{D}^{2} + \sum_{j \in J} μ_{j} + μ_{0} + μ_{\infty} + \sum_{k \in K} {\bar{μ}}_{k} + {\bar{μ}}_{0} + {\bar{μ}}_{\infty}] \\ + (\frac{1}{α + β} - \frac{1}{2_{s_{1}}}) [‖ u ‖_{2_{s_{1}}}^{2_{s_{1}}} + \sum_{j \in J} ν_{j} + ν_{0} + ν_{\infty}] + (\frac{1}{α + β} - \frac{1}{2_{s_{2}}}) [‖ v ‖_{2_{s_{2}}}^{2_{s_{2}}} + \sum_{k \in K} {\bar{ν}}_{j} + {\bar{ν}}_{0} + {\bar{ν}}_{\infty}] \\ ⩾ (\frac{1}{2} - \frac{1}{α + β}) [Λ_{1} \sum_{j} ν_{j}^{2 / 2_{s_{1}}} + Λ_{2} \sum_{k} {\bar{ν}}_{k}^{2 / 2_{s_{2}}} + Λ_{1} ν_{0}^{2 / 2_{s_{1}}} + Λ_{1} ν_{\infty}^{2 / 2_{s_{1}}} + Λ_{2} {\bar{ν}}_{0}^{2 / 2_{s_{2}}} + Λ_{2} {\bar{ν}}_{\infty}^{2 / 2_{s_{2}}}] \\ + (\frac{1}{α + β} - \frac{1}{2_{s_{1}}}) [\sum_{j \in J} ν_{j} + ν_{0} + ν_{\infty}] + (\frac{1}{α + β} - \frac{1}{2_{s_{2}}}) [\sum_{k \in K} {\bar{ν}}_{j} + {\bar{ν}}_{0} + {\bar{ν}}_{\infty}] . \end{aligned}

(3.30)

If the concentration is considered at the point

(x_{j}, y_{j})

only, i.e.

ν_{j} > 0

and further from (3.30) we see that

\begin{aligned} c ⩾ (\frac{1}{2} - \frac{1}{α + β}) [Λ_{1} {(Λ_{1}^{\frac{1 + s_{1}}{2 s_{1}}})}^{\frac{2 (1 - s_{1})}{2 (1 + s_{1})}}] + (\frac{1}{α + β} - \frac{1}{2_{s_{1}}}) Λ_{1}^{\frac{1 + s_{1}}{2 s_{1}}} = (\frac{1}{2} - \frac{1}{2_{s_{1}}}) Λ_{1}^{\frac{1 + s_{1}}{2 s_{1}}} = \frac{s_{1}}{1 + s_{1}} Λ_{1}^{\frac{1 + s_{1}}{2 s_{1}}} . \end{aligned}

(3.31)

From (3.16) we conclude that

ν_{j} = μ_{j} = 0

for all

j \in J

. Arguing similarly we also have

{\bar{ν}}_{k} = {\bar{μ}}_{k} = 0

for all

k \in K

. Now if

ν_{0} \neq 0

, then

\begin{aligned} c ⩾ (\frac{1}{2} - \frac{1}{α + β}) Λ_{1}^{\frac{1 + s_{1}}{2 s_{1}}} + (\frac{1}{α + β} - \frac{1}{2_{s_{1}}}) Λ_{1}^{\frac{1 + s_{1}}{2 s_{1}}} = \frac{s_{1}}{1 + s_{1}} Λ_{1}^{\frac{1 + s_{1}}{2 s_{1}}}, \end{aligned}

which contradicts the assumption (3.16). Therefore,

ν_{0} = 0

. By the same token, we also get

{\bar{ν}}_{0} = 0

. Arguing in a similar way we also get

ν_{\infty} = 0 = {\bar{ν}}_{\infty}

. Hence, there exists a subsequence that strongly converges in

L^{2_{s_{1}}} (R^{2}) \times L^{2_{s_{2}}} (R^{2})

. Furthermore, we have

\begin{aligned} {‖ (u_{n} - u, v_{n} - v) ‖}_{D}^{2} = ⟨ J_{κ}^{^{'}} (u_{n}, v_{n}) | (u_{n} - u, v_{n} - v) ⟩ + o_{n} (1), \end{aligned}

which infers that the sequence

{(u_{n}, v_{n})}

strongly converges in

D

and the (PS) condition holds.

4. .Properties of

{\bar{c}}_{κ}

and proof of Theorem 4

We are devoted to prove Proposition 2 in the first part of this section.

Proof to Proposition 2. Proof to Proposition 2.

Firstly, we prove (1). Let $0 ⩽ κ_{1} < κ_{2}$ . We take a bounded (in $D$ ) minimizing sequence ${(u_{n}, v_{n})}$ converging to $c_{κ_{1}}$ . There exists a unique $η_{n} = η_{(u_{n}, v_{n})}$ such that $(η_{n} u_{n}, η_{n} v_{n}) \in N_{κ_{2}}$ . Thus, we have

\begin{aligned} o_{n} (1) + c_{κ_{1}} = J_{κ_{1}} (u_{n}, v_{n}) & ⩾ J_{κ_{1}} (η_{n} u_{n}, η_{n} v_{n}) = J_{κ_{2}} (η_{n} u_{n}, η_{n} v_{n}) + (κ_{2} - κ_{1}) η_{n}^{α + β} \int_{R^{2}} h (x, y) | u_{n} |^{α} | v_{n} |^{β} d x d y \\ ⩾ J_{κ_{2}} (η_{n} u_{n}, η_{n} v_{n}) ⩾ c_{κ_{2}}, \end{aligned}

where we use (2.6) in the first inequality. Sending n to infinity, one gets

c_{κ_{1}} ⩾ c_{κ_{2}}

. Furthermore, if

c_{κ_{1}}

is achieved by

(u, v)

, we know

u \neq 0

and

v \neq 0

. There exists a unique

η = η_{(u, v)}

such that

(η u, η v) \in N_{κ_{2}}

. Then we have

\begin{aligned} c_{κ_{1}} = J_{κ_{1}} (u, v) & ⩾ J_{κ_{1}} (η u, η v = J_{κ_{2}} (η u, η v) + (κ_{2} - κ_{1}) η^{α + β} \int_{R^{2}} h (x, y) | u |^{α} | v |^{β} d x d y > J_{κ_{2}} (η u, η v) ⩾ c_{κ_{2}} . \end{aligned}

This completes the proof to (1).

Next, we prove (2). By (1), it is sufficient to prove that $c_{κ_{1}} ⩽ \underset{κ \to κ_{1}^{+}}{lim inf} c_{κ}$ for all $κ_{1} ⩾ 0$ . For any sequence ${κ_{n}}$ with $κ_{n} \to κ_{1}^{+}$ , we can take a bounded (in $D$ ) sequence ${(u_{n}, v_{n})}$ such that $J_{κ_{n}} (u_{n}, v_{n}) = c_{κ_{n}} + o_{n} (1)$ . There exists a unique $η_{n} > 0$ such that $(η_{n} u, η_{n} v) \in N_{κ_{1}}$ . We claim that $η_{n}$ is bounded. Indeed, if $η_{n} \to \infty$ up to subsequences, using (2.4) (for $κ = κ_{1}$ ) we have $‖ u_{n} ‖_{2_{s_{1}}} \to 0$ , $‖ v_{n} ‖_{2_{s_{2}}} \to 0$ , $and \int_{R^{2}} h (x, y) | u_{n} |^{α} | v_{n} |^{β} d x d y \to 0$ as $n \to \infty$ . Then using (2.4) again (for $κ = κ_{n}$ ) one gets $‖ (u_{n}, v_{n}) ‖_{D} \to 0$ , which is a contradiction. Hence, $η_{n}$ is bounded. Then we have

\begin{aligned} o_{n} (1) + c_{κ_{n}} = J_{κ_{n}} (u_{n}, v_{n}) & ⩾ J_{κ_{n}} (η_{n} u_{n}, η_{n} v_{n}) = J_{κ_{1}} (η_{n} u_{n}, η_{n} v_{n}) + (κ_{n} - κ_{1}) η_{n}^{α + β} \int_{R^{2}} h (x, y) | u_{n} |^{α} | v_{n} |^{β} d x d y \\ = J_{κ_{1}} (η_{n} u_{n}, η_{n} v_{n}) + o_{n} (1) ⩾ c_{κ_{1}} + o_{n} (1) . \end{aligned}

Sending n to infinity, one gets

\underset{n \to \infty}{lim inf} c_{κ_{n}} ⩾ c_{κ_{1}}

. The proof to (2) is complete.

Finally, we prove (3). We firstly prove that $c_{0} = min {\frac{s_{1}}{1 + s_{1}} Λ_{1}^{\frac{1 + s_{1}}{2 s_{1}}}, \frac{s_{2}}{1 + s_{2}} Λ_{2}^{\frac{1 + s_{2}}{2 s_{2}}}}$ . (2.2) reads in the case $κ = 0$ that for any $(u, v) \in N_{0}$ ,

\begin{aligned} {‖ (u, v) ‖}_{D}^{2} = ‖ u ‖_{2_{s_{1}}}^{2_{s_{1}}} + ‖ v ‖_{2_{s_{2}}}^{2_{s_{2}}} . \end{aligned}

We will consider three cases:

u \neq 0

v = 0

;

u = 0

v \neq 0

; and

u \neq 0

v \neq 0

. When

u \neq 0

v = 0

, from

\begin{aligned} Λ_{1} ‖ u ‖_{2_{s_{1}}}^{2} ⩽ ‖ u ‖_{H^{1, s_{1}} (R^{2})}^{2} = ‖ u ‖_{2_{s_{1}}}^{2_{s_{1}}}, \end{aligned}

we deduce that

‖ u ‖_{2_{s_{1}}} ⩾ Λ_{1}^{\frac{1 - s_{1}}{4 s_{1}}}

. Hence, by using (2.3) one gets

\begin{aligned} J_{κ} (u, 0) = \frac{s_{1}}{1 + s_{1}} ‖ u ‖_{2_{s_{1}}}^{2_{s_{1}}} ⩾ \frac{s_{1}}{1 + s_{1}} Λ_{1}^{\frac{1 + s_{1}}{2 s_{1}}} . \end{aligned}

Similarly, for any

(0, v) \in N_{0}

, we have

\begin{aligned} J_{κ} (0, v) ⩾ \frac{s_{2}}{1 + s_{2}} Λ_{2}^{\frac{1 + s_{2}}{2 s_{2}}} . \end{aligned}

When

u \neq 0

v \neq 0

, from

\begin{aligned} Λ_{1} ‖ u ‖_{2_{s_{1}}}^{2} + Λ_{2} ‖ v ‖_{2_{s_{2}}}^{2} ⩽ {‖ (u, v) ‖}_{D}^{2} = ‖ u ‖_{2_{s_{1}}}^{2_{s_{1}}} + ‖ v ‖_{2_{s_{2}}}^{2_{s_{2}}}, \end{aligned}

we deduce that

Λ_{1} ‖ u ‖_{2_{s_{1}}}^{2} ⩽ ‖ u ‖_{2_{s_{1}}}^{2_{s_{1}}}

Λ_{2} ‖ v ‖_{2_{s_{2}}}^{2} ⩽ ‖ v ‖_{2_{s_{2}}}^{2_{s_{2}}}

. If the former holds, we have

\begin{aligned} J_{κ} (u, v) = \frac{s_{1}}{1 + s_{1}} ‖ u ‖_{2_{s_{1}}}^{2_{s_{1}}} + \frac{s_{2}}{1 + s_{2}} ‖ v ‖_{2_{s_{2}}}^{2_{s_{2}}} > \frac{s_{1}}{1 + s_{1}} Λ_{1}^{\frac{1 + s_{1}}{2 s_{1}}} . \end{aligned}

If the later holds, we have

\begin{aligned} J_{κ} (u, v) = \frac{s_{1}}{1 + s_{1}} ‖ u ‖_{2_{s_{1}}}^{2_{s_{1}}} + \frac{s_{2}}{1 + s_{2}} ‖ v ‖_{2_{s_{2}}}^{2_{s_{2}}} > \frac{s_{2}}{1 + s_{2}} Λ_{2}^{\frac{1 + s_{2}}{2 s_{2}}} . \end{aligned}

Then we can deduce that

c_{0} ⩾ min {\frac{s_{1}}{1 + s_{1}} Λ_{1}^{\frac{1 + s_{1}}{2 s_{1}}}, \frac{s_{2}}{1 + s_{2}} Λ_{2}^{\frac{1 + s_{2}}{2 s_{2}}}}

. On the other hand, WLOG, we assume that

\frac{s_{1}}{1 + s_{1}} Λ_{1}^{\frac{1 + s_{1}}{2 s_{1}}} ⩽ \frac{s_{2}}{1 + s_{2}} Λ_{2}^{\frac{1 + s_{2}}{2 s_{2}}}

. Take a minimizing sequence

{u_{n}}

Λ_{1}

, i.e.,

\frac{‖ u_{n} ‖_{H^{1, s_{1}} (R^{2})}^{2}}{‖ u_{n} ‖_{2_{s_{1}}}^{2}} \to Λ_{1}

n \to \infty

. By replacing by

{η_{n} u_{n}}

(also denoted by

{u_{n}}

) for some

{η_{n}} \subset R_{+}

, we may assume that

{(u_{n}, 0)} \in N_{0}

. Then

\begin{aligned} J_{κ} (u_{n}, 0) = \frac{s_{1}}{1 + s_{1}} ‖ u_{n} ‖_{2_{s_{1}}}^{2_{s_{1}}} \to \frac{s_{1}}{1 + s_{1}} Λ_{1}^{\frac{1 + s_{1}}{2 s_{1}}} as n \to \infty, \end{aligned}

implying that

c_{0} ⩽ \frac{s_{1}}{1 + s_{1}} Λ_{1}^{\frac{1 + s_{1}}{2_{s_{1}}}}

. Hence, we obtain that

c_{0} = min {\frac{s_{1}}{1 + s_{1}} Λ_{1}^{\frac{1 + s_{1}}{2 s_{1}}}, \frac{s_{2}}{1 + s_{2}} Λ_{2}^{\frac{1 + s_{2}}{2 s_{2}}}}

Further, We prove that ${\bar{c}}_{κ} < min {\frac{s_{1}}{1 + s_{1}} Λ_{1}^{\frac{1 + s_{1}}{2 s_{1}}}, \frac{s_{2}}{1 + s_{2}} Λ_{2}^{\frac{1 + s_{2}}{2 s_{2}}}}$ for κ large enough. For any $(u, v) \in D ∖ {(0, 0)}$ , there exists a unique $t = t_{(u, v)} > 0$ such that $(t u, t v) \in N_{κ}$ and satisfies the algebraic equation

\begin{aligned} {‖ (u, v) ‖}_{D}^{2} & = t^{2_{s_{1}} - 2} ‖ u ‖_{2_{s_{1}}}^{2_{s_{1}}} + t^{2_{s_{2}} - 2} ‖ v ‖_{2_{s_{2}}}^{2_{s_{2}}} + κ (α + β) t^{α + β - 2} \int_{R^{2}} h (x, y) | u |^{α} | v |^{β} d x d y . \\ t^{α + β - 2} & = \frac{‖ (u, v) ‖_{D}^{2} - t^{2_{s_{1}} - 2} ‖ u ‖_{2_{s_{1}}}^{2_{s_{1}}} - t^{2_{s_{2}} - 2} ‖ v ‖_{2_{s_{2}}}^{2_{s_{2}}}}{κ (α + β) \int_{R^{2}} h (x, y) | u |^{α} | v |^{β} d x d y} . \end{aligned}

Suppose

κ \to + \infty

, then using the assumption

α + β > 2

we get

t = t_{κ} \to 0

. Subsequently, we can write

\begin{aligned} lim_{κ \to + \infty} t_{κ}^{α + β - 2} κ & = lim_{κ \to + \infty} \frac{‖ (u, v) ‖_{D}^{2} - t_{κ}^{2_{s_{1}} - 2} ‖ u ‖_{2_{s_{1}}}^{2_{s_{1}}} - t_{κ}^{2_{s_{2}} - 2} ‖ v ‖_{2_{s_{2}}}^{2_{s_{2}}}}{(α + β) \int_{R^{2}} h (x, y) | u |^{α} | v |^{β} d x d y}, \\ lim_{κ \to + \infty} t_{κ}^{α + β - 2} κ & = \frac{‖ (u, v) ‖_{D}^{2}}{(α + β) \int_{R^{2}} h (x, y) | u |^{α} | v |^{β} d x d y}, since t_{κ} \to 0 as κ \to + \infty . \end{aligned}

Further we check the behaviour of the functional

J_{κ} (t_{κ} u, t_{κ} v)

κ \to + \infty

. The functional is given by

\begin{aligned} J_{κ} (t_{κ} u, t_{κ} v) & = \frac{t_{κ}^{2}}{2} {‖ (u, v) ‖}_{D}^{2} - \frac{t_{κ}^{2_{s_{1}}}}{2_{s_{1}}} \int_{R^{2}} | u |^{2_{s_{1}}} d x d y - \frac{t_{κ}^{2_{s_{2}}}}{2_{s_{2}}} \int_{R^{2}} | v |^{2_{s_{2}}} d x d y - κ t_{κ}^{α + β} \int_{R^{2}} h (x, y) | u |^{α} | v |^{β} d x d y \\ = \frac{t_{κ}^{2}}{2} {‖ (u, v) ‖}_{D}^{2} - \frac{t_{κ}^{2_{s_{1}}}}{2_{s_{1}}} \int_{R^{2}} | u |^{2_{s_{1}}} d x d y - \frac{t_{κ}^{2_{s_{2}}}}{2_{s_{2}}} \int_{R^{2}} | v |^{2_{s_{2}}} d x d y - \frac{1}{α + β} t_{κ}^{2} {‖ (u, v) ‖}_{D}^{2} \\ + \frac{t_{κ}^{2_{s_{1}}}}{α + β} \int_{R^{2}} | u |^{2_{s_{1}}} d x d y + \frac{t_{κ}^{2_{s_{2}}}}{α + β} \int_{R^{2}} | v |^{2_{s_{2}}} d x d y \\ = (\frac{1}{2} - \frac{1}{α + β}) t_{κ}^{2} {‖ (u, v) ‖}_{D}^{2} + (\frac{1}{α + β} - \frac{1}{2_{s_{1}}}) t_{κ}^{2_{s_{1}}} \int_{R^{2}} | u |^{2_{s_{1}}} d x d y \\ + (\frac{1}{α + β} - \frac{1}{2_{s_{2}}}) t_{κ}^{2_{s_{2}}} \int_{R^{2}} | v |^{2_{s_{2}}} d x d y \\ = (\frac{1}{2} - \frac{1}{α + β} + A_{κ} (u, v)) t_{κ}^{2} {‖ (u, v) ‖}_{D}^{2} . \end{aligned}

The term

A_{κ} (u, v)

is given by

\begin{aligned} A_{κ} (u, v) = \frac{t_{κ}^{2_{s_{1}} - 2}}{‖ (u, v) ‖_{D}^{2}} (\frac{1}{α + β} - \frac{1}{2_{s_{1}}}) \int_{R^{2}} | u |^{2_{s_{1}}} d x d y + \frac{t_{κ}^{2_{s_{2}} - 2}}{‖ (u, v) ‖_{D}^{2}} (\frac{1}{α + β} - \frac{1}{2_{s_{2}}}) \int_{R^{2}} | v |^{2_{s_{2}}} d x d y . \end{aligned}

Notice that

A_{κ} (u, v) \to 0

κ \to + \infty

. Hence,

J_{κ} (t_{κ} u, t_{κ} v) = o_{κ} (1)

κ \to + \infty

and there exists a

κ_{0} > 0

such that if

κ > κ_{0}

, where

κ_{0}

is sufficiently large, then

J_{κ} (t_{κ} u, t_{κ} v) < min {\frac{s_{1}}{1 + s_{1}} Λ_{1}^{\frac{1 + s_{1}}{2 s_{1}}}, \frac{s_{2}}{1 + s_{2}} Λ_{2}^{\frac{1 + s_{2}}{2 s_{2}}}}

. Thus we obtain

\begin{aligned} {\bar{c}}_{κ} = inf_{(u, v) \in N_{κ}} J_{κ} (u, v) ⩽ J_{κ} (t_{κ} u, t_{κ} v) < min {\frac{s_{1}}{1 + s_{1}} Λ_{1}^{\frac{1 + s_{1}}{2 s_{1}}}, \frac{s_{2}}{1 + s_{2}} Λ_{2}^{\frac{1 + s_{2}}{2 s_{2}}}} \end{aligned}

and hence from the above inequality we get

\begin{aligned} {\bar{c}}_{κ} < min {\frac{s_{1}}{1 + s_{1}} Λ_{1}^{\frac{1 + s_{1}}{2 s_{1}}}, \frac{s_{2}}{1 + s_{2}} Λ_{2}^{\frac{1 + s_{2}}{2 s_{2}}}} . \end{aligned}

Now we know that there exists $κ^{*} \in (0, \infty)$ such that

\begin{aligned} {\bar{c}}_{κ} & = min {\frac{s_{1}}{1 + s_{1}} Λ_{1}^{\frac{1 + s_{1}}{2 s_{1}}}, \frac{s_{2}}{1 + s_{2}} Λ_{2}^{\frac{1 + s_{2}}{2 s_{2}}}}, \forall κ \in [0, κ^{*}), \\ {\bar{c}}_{κ} & < min {\frac{s_{1}}{1 + s_{1}} Λ_{1}^{\frac{1 + s_{1}}{2 s_{1}}}, \frac{s_{2}}{1 + s_{2}} Λ_{2}^{\frac{1 + s_{2}}{2 s_{2}}}}, \forall κ > κ^{*} . \end{aligned}

Since

{\bar{c}}_{κ}

is continuous, we have

\begin{aligned} {\bar{c}}_{κ^{*}} = min {\frac{s_{1}}{1 + s_{1}} Λ_{1}^{\frac{1 + s_{1}}{2 s_{1}}}, \frac{s_{2}}{1 + s_{2}} Λ_{2}^{\frac{1 + s_{2}}{2 s_{2}}}} . \end{aligned}

The proof is complete.

In the rest part of this section, we prove Theorem 4.

Proof of Theorem 4. Proof of Theorem 4.

We firstly consider the case that $κ > κ^{*}$ and prove (1). By Proposition 2-(3),

\begin{aligned} {\bar{c}}_{κ} < min {\frac{s_{1}}{1 + s_{1}} Λ_{1}^{\frac{1 + s_{1}}{2 s_{1}}}, \frac{s_{2}}{1 + s_{2}} Λ_{2}^{\frac{1 + s_{2}}{2 s_{2}}}}, \end{aligned}

for any

κ > κ^{*}

. Then we use Lemma 22 to ensure the existence of

(u_{0}, v_{0}) \in D

such that

{\bar{c}}_{κ} = J_{κ} (u_{0}, v_{0})

. Next, we define the function

ψ (t) : (0, \infty) \to R

given by

ψ (t) = J_{κ} (t u, t v)

, for all

t > 0

. Then

\begin{aligned} ψ (t) & = \frac{t^{2}}{2} A_{1} - \frac{t^{2_{s_{1}}}}{2_{s_{1}}} A_{2} - \frac{t^{2_{s_{2}}}}{2_{s_{2}}} A_{3} - A_{4} κ t^{α + β}, \\ ψ^{'} (t) & = A_{1} t - A_{2} t^{2_{s_{1}} - 1} - A_{3} t^{2_{s_{2}} - 1} - A_{4} κ (α + β) t^{α + β - 1}, \\ ψ^{″} (t) & = A_{1} - A_{2} (2_{s_{1}} - 1) t^{2_{s_{1}} - 2} - A_{3} (2_{s_{2}} - 1) t^{2_{s_{2}} - 2} - A_{4} κ (α + β) (α + β - 1) t^{α + β - 2}, \\ ψ^{‴} (t) & = - A_{2} (2_{s_{1}} - 1) (2_{s_{1}} - 2) t^{2_{s_{1}} - 3} - A_{3} (2_{s_{2}} - 1) (2_{s_{2}} - 2) t^{2_{s_{2}} - 3} \\ - A_{4} κ (α + β) (α + β - 1) (α + β - 2) t^{α + β - 3}, \end{aligned}

where

A_{1} = ‖ (u, v) ‖_{D}^{2}

A_{2} = ‖ u ‖_{2_{s_{1}}}^{2_{s_{1}}}

A_{3} = ‖ v ‖_{2_{s_{2}}}^{2_{s_{2}}}

A_{4} = \int_{R^{2}} h (x, y) | u |^{α} | v |^{β} d x d y

. Clearly,

A_{1}

A_{2}

A_{3} and A_{4}

are non-negative. The condition

α + β > 2

implies that

ψ^{‴} (t) < 0

for all t > 0

. Therefore the function

ψ^{'} (t)

is strictly concave for

t > 0

. Also, we have

lim_{t \to 0} ψ^{'} (t) = 0

and

lim_{t \to + \infty} ψ^{'} (t) = - \infty

. Moreover, the function

ψ^{'} (t) > 0

for

t > 0

small enough. Hence,

ψ^{'} (t)

has a unique global maximum point at

t = t_{0}

and

ψ^{'} (t)

has a unique root at

t_{1} > t_{0}

and

ψ^{″} (t) < 0

for

t > t_{0}

, in particular

ψ^{″} (t_{1}) < 0

. Also, from the equation (2.5) and

ψ (t)

, we observe that

(t u, t v) \in N_{κ}

if and only if

ψ^{″} (t) < 0

Now we consider the function $(| u_{0} |, | v_{0} |) \in D$ , then from the above arguments there exists a unique $t_{2} > 0$ such that $t_{2} (| u_{0} |, | v_{0} |) = (t_{2} | u_{0} |, t_{2} | v_{0} |) \in N_{κ}$ and $t_{2}$ satisfies the following algebraic equation

\begin{aligned} {‖ (| u_{0} |, | v_{0} |) ‖}_{D}^{2} = t_{2}^{2_{s_{1}} - 2} ‖ u_{0} ‖_{2_{s_{1}}}^{2_{s_{1}}} + t_{2}^{2_{s_{2}} - 2} ‖ v_{0} ‖_{2_{s_{2}}}^{2_{s_{2}}} + κ (α + β) t_{2}^{α + β - 2} \int_{R^{2}} h (x, y) | u_{0} |^{α} | v_{0} |^{β} d x d y . \end{aligned}

Moreover, the pair

(u_{0}, v_{0}) \in N_{κ}

gives

\begin{aligned} {‖ (u_{0}, v_{0}) ‖}_{D}^{2} = ‖ u_{0} ‖_{2_{s_{1}}}^{2_{s_{1}}} + ‖ v_{0} ‖_{2_{s_{2}}}^{2_{s_{2}}} + κ (α + β) \int_{R^{2}} h (x, y) | u_{0} |^{α} | v_{0} |^{β} d x d y . \end{aligned}

Now from the inequality

‖ (| u_{0} |, | v_{0} |) ‖_{D} ⩽ ‖ (u_{0}, v_{0}) ‖_{D}

, one finds that

t_{2} ⩽ 1

. Since

(u_{0}, v_{0}) \in N_{κ}

and

(u_{0}, v_{0})

is the unique maximum point of

ψ (t) = J_{κ} (t u_{0}, t v_{0})

\forall t > t_{0}

. We can deduce that

\begin{aligned} {\bar{c}}_{κ} = J_{κ} (u_{0}, v_{0}) = max_{t > t_{0}} J_{κ} (t u_{0}, t v_{0}) ⩾ J_{κ} (t_{2} u_{0}, t_{2} v_{0}) ⩾ J_{κ} (t_{2} | u_{0} |, t_{2} | v_{0} |) ⩾ {\bar{c}}_{κ} . \end{aligned}

Thus, we can assume that

u_{0} ⩾ 0

and

v_{0} ⩾ 0

R^{2}

. If

u_{0} \equiv 0

, then

v_{0}

satisfies the equation

\begin{aligned} - \partial_{x x} v + {(- Δ)}_{y}^{s} v + v = v^{2_{s_{2}} - 1} in R^{2}, \end{aligned}

(4.1)

which implies

v_{0} \equiv 0

, since the above equation has only trivial solution (see [7, Lemma 2.2]). Therefore, we get a contradiction to the fact that

(u_{0}, v_{0}) \in N_{k}

. So we can conclude that either

u_{0}

v_{0}

is not identically zero.

Notice that the Nehari manifold $N_{κ}$ consists of all the non-trivial critical points of the functional $J_{κ}$ . This implies that $c_{κ} ⩾ {\bar{c}}_{κ}$ . On the other hand, $c_{κ} ⩽ J_{κ} (u_{0}, v_{0}) = {\bar{c}}_{κ}$ . Hence, $c_{κ} = {\bar{c}}_{κ}$ , and so $(u_{0}, v_{0})$ is a non-negative (constant sign) ground state solution to the system (1.1).

Next, we consider the case that $κ \in [0, κ^{*})$ and prove (2). Suppose on the contrary that there exists a $κ \in [0, κ^{*})$ such that $c_{κ}$ has a minimizer. By Proposition 2-(1), we have $c_{κ^{*}} < c_{κ}$ , which is a contradiction to Proposition 2-(3). The proof is complete.

Remark 23.

Notice that we have derived only the non-negative solutions and to obtain a positive solution we require the maximum principle for the mixed type Schrödinger operator.

5. .Proofs of Theorem 5

The proof of Proposition 7 is similar to the one of Proposition 2 and we omit the details.

To prove Theorem 5, we firstly give a preliminary compactness result in the critical case. Recall that (H1) is given as

\begin{aligned} 0 ⩽ h \in L^{1} (R^{2}) \cap L^{\infty} (R^{2}), h is not zero, continuous near 0 and \infty, and \\ h (0, 0) = lim_{| (x, y) | \to + \infty} h (x, y) = 0. \end{aligned}

(H1)

and the space of radial functions in

D

is defined as

\begin{aligned} D_{r} := H_{r}^{1, s} (R^{2}) \times H_{r}^{1, s} (R^{2}) = {(u, v) \in D : u and v are radially symmetric} . \end{aligned}

Lemma 24.

If $α + β = 2_{s}$ and h is a radial function satisfying (H1). If ${(u_{n}, v_{n})} \subset D_{r}$ is a (PS) sequence for $J_{κ}$ at level $c \in R$ such that c satisfies

\begin{aligned} c < \frac{s}{1 + s} Λ_{r}^{\frac{1 + s}{2 s}}, \end{aligned}

(5.1)

the sequence

{(u_{n}, v_{n})}

admits a strongly converging subsequence in

D

Proof.

Since the functions ${(u_{n}, v_{n})} \subset D_{r}$ are radial, having concentrations at points other than $0 or \infty$ gives a contradiction to countability of concentration points. Now if we want to avoid concentration at the points 0 and ∞, it is sufficient to show that

\begin{aligned} lim_{ϵ \to 0} \underset{n \to + \infty}{lim sup} \int_{R^{2}} h (x, y) | u_{n} |^{α} | v_{n} |^{β} Ψ_{0, ϵ} (x, y) d x d y = 0, \end{aligned}

(5.2)

\begin{aligned} lim_{R \to + \infty} \underset{n \to + \infty}{lim sup} \int_{\sqrt{x^{2} + y^{2}} > R} h (x, y) | u_{n} |^{α} | v_{n} |^{β} Ψ_{\infty, ϵ} (x, y) d x d y = 0, \end{aligned}

(5.3)

where the cut-off function

Ψ_{0, ϵ}

is centred at

(0, 0)

satisfying (3.20) and the cut-off function

Ψ_{\infty, ϵ}

supported near ∞ satisfying the following for

R > 0

large enough

\begin{aligned} Ψ_{\infty, ϵ} = 0 in B_{R} (0, 0), Ψ_{\infty, ϵ} = 1 in B_{R + 1}^{c} (0, 0), 0 ⩽ Ψ_{\infty, ϵ} ⩽ 1 and | \nabla Ψ_{\infty, ϵ} | ⩽ \frac{4}{ϵ} . \end{aligned}

(5.4)

For any $s \in (0, 1)$ , the assumption $α + β = 2_{s}$ implies that $\frac{α}{2_{s}} + \frac{β}{2_{s}} = 1$ . Using the assumption on h in (H1) and the Hölder’s inequality, we get

\begin{aligned} \int_{R^{2}} h (x, y) | u_{n} |^{α} | v_{n} |^{β} Ψ_{0, ϵ} (x, y) d x d y & = \int_{R^{2}} {(h (x, y) Ψ_{0, ϵ} (x, y))}^{\frac{α}{2_{s}} + \frac{β}{2_{s}}} | u_{n} |^{α} | v_{n} |^{β} d x d y \\ ⩽ {(\int_{R^{2}} h (x, y) | u_{n} |^{2_{s}} Ψ_{0, ϵ} (x, y) d x d y)}^{\frac{α}{2_{s}}} {(\int_{R^{2}} h (x, y) | v_{n} |^{2_{s}} Ψ_{0, ϵ} (x, y) d x d y)}^{\frac{β}{2_{s}}} . \end{aligned}

Now from (3.17) and (H1), we have

\begin{aligned} lim_{n \to + \infty} \int_{R^{2}} h (x, y) | u_{n} |^{2_{s}} Ψ_{0, ϵ} (x, y) d x d y & = \int_{R^{2}} h (x, y) | u_{0} |^{2_{s}} Ψ_{0, ϵ} (x, y) d x d y + ν_{0} h (0, 0) \\ ⩽ \int_{| (x, y) | ⩽ ϵ} h (x, y) | u_{0} |^{2_{s}} d x d y, since h (0, 0) = 0 \end{aligned}

and

\begin{aligned} lim_{n \to + \infty} \int_{R^{2}} h (x, y) | v_{n} |^{2_{s}} Ψ_{0, ϵ} (x, y) d x d y & = \int_{R^{2}} h (x, y) | v_{0} |^{2_{s}} Ψ_{0, ϵ} (x, y) d x d y + {\bar{ν}}_{0} h (0, 0) \\ ⩽ \int_{| (x, y) | ⩽ ϵ} h (x, y) | v_{0} |^{2_{s}} d x d y, since h (0, 0) = 0. \end{aligned}

From the above three inequalities, it follows that

\begin{aligned} lim_{ϵ \to 0} \underset{n \to + \infty}{lim sup} \int_{R^{2}} h (x, y) | u_{n} |^{α} | v_{n} |^{β} Ψ_{0, ϵ} (x, y) d x d y \\ ⩽ lim_{ϵ \to 0} [{(\int_{| (x, y) | ⩽ ϵ} h (x, y) | u_{0} |^{2_{s}} d x d y)}^{\frac{α}{2_{s}}} {(\int_{| (x, y) | ⩽ ϵ} h (x, y) | v_{0} |^{2_{s}} d x d y)}^{\frac{β}{2_{s}}}] = 0. \end{aligned}

Similarly, we can prove (5.3) using the assumption that

lim_{| (x, y) | \to + \infty} h (x, y) = 0

Then using similar arguments to Lemma 22, we can complete the proof.

Proof of Theorem 5. Proof of Theorem 5.

By Proposition 7-(3),

\begin{aligned} {\bar{c}}_{κ, r} < \frac{s}{1 + s} Λ_{r}^{\frac{1 + s}{2 s}}, \end{aligned}

for any

κ > κ^{* *}

. We use Lemma 24 to ensure the existence of

(u_{0}, v_{0}) \in D_{r}

such that

{\bar{c}}_{κ, r} = J_{κ} (u_{0}, v_{0})

. Then similar to the proof of Theorem 4, we know

(u_{0}, v_{0})

is a non-negative (constant sign) non-trivial solution to the system (1.1). The proof is complete.

6. .Proof of Theorem 8

In the following we prove the Pohozaev type identity which will be useful to derive the non-existence results to the system (1.1).

Lemma 25.
Assume that $(u, v) \in D$ is a solution to (1.1) such that $x u, y u \in L^{2} (R^{2})$ , $h \in L^{\infty} (R^{2}) \cap C^{1} (R^{2})$ , and $α + β = 2_{s}$ . Then $(u, v)$ satisfies the following necessary conditions:
$\begin{aligned} s \int_{R^{2}} (| \partial_{x} u |^{2} + | \partial_{x} v |^{2} + {| {(- Δ)}_{y}^{s / 2} u |}^{2} + {| {(- Δ)}_{y}^{s / 2} v |}^{2}) d x d y & = s \int_{R^{2}} (| u |^{2_{s}} + | v |^{2_{s}}) d x d y + κ s 2_{s} \int_{R^{2}} h (x, y) | u |^{α} | v |^{β} d x d y \\ - κ \int_{R^{2}} (y h_{y} + s x h_{x}) | u |^{α} | v |^{β} d x d y \end{aligned}$
(6.1)
and
$\begin{aligned} \int_{R^{2}} (| \partial_{x} u |^{2} + | \partial_{x} v |^{2} + {| {(- Δ)}_{y}^{s / 2} u |}^{2} + {| {(- Δ)}_{y}^{s / 2} v |}^{2} + | u |^{2} + | v |^{2}) d x d y \\ = \int_{R^{2}} (| u |^{2_{s}} + | v |^{2_{s}}) d x d y + κ 2_{s} \int_{R^{2}} h (x, y) | u |^{α} | v |^{β} d x d y . \end{aligned}$
(6.2)
Proof.
Let $F (u)$ denotes the Fourier transform of u defined by
$\begin{aligned} F (u) (ξ_{1}, ξ_{2}) := \int_{R^{2}} u (x, y) e^{- 2 π i (x, y) \cdot (ξ_{1}, ξ_{2})} d x d y \end{aligned}$
and $F^{- 1}$ denotes the inverse Fourier transform of u. Then we observe that
$\begin{aligned} x \partial_{x} u = - F^{- 1} (e^{- 2 π i ξ_{1}} F (x u) + F u), y \partial_{y} u = - F^{- 1} (e^{- 2 π i ξ_{2}} F (y u) + F u) . \end{aligned}$
(6.3)
It follows from (6.3) that $x \partial_{x} u, y \partial_{y} u \in L^{2} (R^{2})$ using $x u, y u \in L^{2} (R^{2})$ .

We multiply the first equation in (1.1) by $y \partial_{y} u$ and then integrate over $R^{2}$ to obtain the following:
$\begin{aligned} - \int_{R^{2}} y \partial_{x x} u \partial_{y} u d x d y + \int_{R^{2}} {(- Δ)}_{y}^{s} u (y \partial_{y} u) d x d y + \int_{R^{2}} y u \partial_{y} u d x d y \\ = \int_{R^{2}} y | u |^{2_{s} - 2} u \partial_{y} u d x d y + κ α \int_{R^{2}} y h (x, y) | u |^{α - 2} u | v |^{β} \partial_{y} u d x d y . \end{aligned}$
(6.4)
Now we separately simplify each integral in the above equation. From Gauss-Divergence theorem, we obtain
$\begin{aligned} - \int_{R^{2}} y \partial_{x x} u \partial_{y} u d x d y & = \int_{R^{2}} y \partial_{x} u \partial_{x y} u d x d y = \frac{1}{2} \int_{R^{2}} y \partial_{y} | \partial_{x} u |^{2} d x d y \\ = - \frac{1}{2} \int_{R^{2}} | \partial_{x} u |^{2} d x d y . \end{aligned}$
(6.5)

$\begin{aligned} \int_{R^{2}} y u \partial_{y} u d x d y & = \frac{1}{2} \int_{R^{2}} y \partial_{y} | u |^{2} d x d y = - \frac{1}{2} \int_{R^{2}} | u |^{2} d x d y . \end{aligned}$
(6.6)

$\begin{aligned} \int_{R^{2}} y | u |^{2_{s} - 2} u \partial_{y} u d x d y & = \frac{1}{2_{s}} \int_{R^{2}} y \partial_{y} | u |^{2_{s}} d x d y = - \frac{1}{2_{s}} \int_{R^{2}} | u |^{2_{s}} d x d y . \end{aligned}$
(6.7)

$\begin{aligned} \int_{R^{2}} y h (x, y) | u |^{α - 2} u | v |^{β} \partial_{y} u d x d y & = \frac{1}{α} \int_{R^{2}} y h (x, y) \partial_{y} | u |^{α} | v |^{β} d x d y \\ = - \frac{1}{α} \int_{R^{2}} (y h_{y} (x, y) + h (x, y)) | u |^{α} | v |^{β} d x d y - \frac{β}{α} \int_{R^{2}} y h (x, y) | v |^{β - 2} v (\partial_{y} v) | u |^{α} d x d y . \end{aligned}$
(6.8)
Note that $[{(- Δ)}_{y}^{s}, y \partial_{y}] = 2 s {(- Δ)}_{y}^{s}$ and ${(- Δ)}_{y}^{s} (y \partial_{y} u) = 2 s {(- Δ)}_{y}^{s} u + y \partial_{y} ({(- Δ)}_{y}^{s} u)$ . Using the previous identities, we further evaluate the following integral.
$\begin{aligned} \int_{R^{2}} {(- Δ)}_{y}^{s} u (y \partial_{y} u) d x d y = \int_{R^{2}} u {(- Δ)}_{y}^{s} (y \partial_{y} u) d x d y & = 2 s \int_{R^{2}} u {(- Δ)}_{y}^{s} u d x d y + \int_{R^{2}} y u \partial_{y} {(- Δ)}_{y}^{s} u d x d y \\ = (2 s - 1) \int_{R^{2}} u {(- Δ)}_{y}^{s} u d x d y - \int_{R^{2}} y \partial_{y} u {(- Δ)}_{y}^{s} u d x d y . \end{aligned}$
From above, we deduce that
$\begin{aligned} \int_{R^{2}} {(- Δ)}_{y}^{s} u (y \partial_{y} u) d x d y = \frac{2 s - 1}{2} \int_{R^{2}} u {(- Δ)}_{y}^{s} u d x d y = - \frac{1 - 2 s}{2} \int_{R^{2}} {| {(- Δ)}_{y}^{s / 2} u |}^{2} d x d y . \end{aligned}$
(6.9)
From equations (6.4)–(6.9), we obtain
$\begin{aligned} - \frac{1}{2} \int_{R^{2}} | \partial_{x} u |^{2} d x d y - \frac{1 - 2 s}{2} \int_{R^{2}} {| {(- Δ)}_{y}^{s / 2} u |}^{2} d x d y - \frac{1}{2} \int_{R^{2}} | u |^{2} d x d y \\ = - \frac{1}{2_{s}} \int_{R^{2}} | u |^{2_{s}} d x d y - κ \int_{R^{2}} (y h_{y} (x, y) + h (x, y)) | u |^{α} | v |^{β} d x d y \\ - κ β \int_{R^{2}} y h (x, y) | v |^{β - 2} v (\partial_{y} v) | u |^{α} d x d y . \end{aligned}$
We write the above equation as
$\begin{aligned} \frac{1}{2} \int_{R^{2}} | \partial_{x} u |^{2} d x d y + \frac{1 - 2 s}{2} \int_{R^{2}} | {(- Δ)}_{y}^{s / 2} u |^{2} d x d y + \frac{1}{2} \int_{R^{2}} | u |^{2} d x d y \\ = \frac{1}{2_{s}} \int_{R^{2}} | u |^{2_{s}} d x d y + κ \int_{R^{2}} (y h_{y} (x, y) + h (x, y)) | u |^{α} | v |^{β} d x d y \\ + κ β \int_{R^{2}} y h (x, y) | v |^{β - 2} v (\partial_{y} v) | u |^{α} d x d y . \end{aligned}$
(6.10)

Similarly, multiplying the second equation in (1.1) by $y \partial_{y} v$ and integrating over $R^{2}$ , we have
$\begin{aligned} \frac{1}{2} \int_{R^{2}} | \partial_{x} v |^{2} d x d y + \frac{1 - 2 s}{2} \int_{R^{2}} {| {(- Δ)}_{y}^{s / 2} v |}^{2} d x d y + \frac{1}{2} \int_{R^{2}} | v |^{2} d x d y \\ = \frac{1}{2_{s}} \int_{R^{2}} | v |^{2_{s}} d x d y + κ \int_{R^{2}} (y h_{y} (x, y) + h (x, y)) | u |^{α} | v |^{β} d x d y \\ + κ α \int_{R^{2}} y h (x, y) | u |^{α - 2} u (\partial_{y} u) | v |^{β} d x d y . \end{aligned}$
(6.11)

On the other hand, when we multiply the first equation in (1.1) by $x \partial_{x} u$ and integrating over $R^{2}$ , we have the following equation
$\begin{aligned} - \int_{R^{2}} x \partial_{x x} u \partial_{x} u d x d y + \int_{R^{2}} {(- Δ)}_{y}^{s} u (x \partial_{x} u) d x d y + \int_{R^{2}} x u \partial_{x} u d x d y \\ = \int_{R^{2}} x | u |^{2_{s} - 2} u \partial_{x} u d x d y + κ α \int_{R^{2}} x h (x, y) | u |^{α - 2} u | v |^{β} \partial_{x} u d x d y . \end{aligned}$
(6.12)
Using similar arguments as done to obtain (6.10), we get
$\begin{aligned} - \frac{1}{2} \int_{R^{2}} | \partial_{x} u |^{2} d x d y + \frac{1}{2} \int_{R^{2}} {| {(- Δ)}_{y}^{s / 2} u |}^{2} d x d y + \frac{1}{2} \int_{R^{2}} | u |^{2} d x d y \\ = \frac{1}{2_{s}} \int_{R^{2}} | u |^{2_{s}} d x d y + κ \int_{R^{2}} (x h_{x} (x, y) + h (x, y)) | u |^{α} | v |^{β} d x d y \\ + κ β \int_{R^{2}} x h (x, y) | v |^{β - 2} v (\partial_{x} v) | u |^{α} d x d y . \end{aligned}$
(6.13)
Similarly, multiplying the second equation in (1.1) by $x \partial_{x} v$ , we obtain
$\begin{aligned} - \frac{1}{2} \int_{R^{2}} | \partial_{x} v |^{2} d x d y + \frac{1}{2} \int_{R^{2}} {| {(- Δ)}_{y}^{s / 2} v |}^{2} d x d y + \frac{1}{2} \int_{R^{2}} | v |^{2} d x d y \\ = \frac{1}{2_{s}} \int_{R^{2}} | v |^{2_{s}} d x d y + κ \int_{R^{2}} (x h_{x} (x, y) + h (x, y)) | u |^{α} | v |^{β} d x d y \\ + κ α \int_{R^{2}} x h (x, y) | u |^{α - 2} u (\partial_{x} u) | v |^{β} d x d y . \end{aligned}$
(6.14)
Multiplying (6.13) by s and adding into (6.10) give
$\begin{aligned} \frac{1 - s}{2} \int_{R^{2}} | \partial_{x} u |^{2} d x d y + \frac{1 - s}{2} \int_{R^{2}} {| {(- Δ)}_{y}^{s / 2} u |}^{2} d x d y + \frac{1 + s}{2} \int_{R^{2}} | u |^{2} d x d y \\ = \frac{1 + s}{2_{s}} \int_{R^{2}} | u |^{2_{s}} d x d y + κ (1 + s) \int_{R^{2}} h (x, y) | u |^{α} | v |^{β} d x d y \\ + κ \int_{R^{2}} (y h_{y} (x, y) + s x h_{x} (x, y)) | u |^{α} | v |^{β} d x d y \\ + κ β \int_{R^{2}} (y h (x, y) | v |^{β - 2} v (\partial_{y} v) + s x h | v |^{β - 2} v \partial_{x} v) | u |^{α} d x d y . \end{aligned}$
(6.15)
Similarly, multiplying (6.14) by s and adding into (6.11) gives
$\begin{aligned} \frac{1 - s}{2} \int_{R^{2}} | \partial_{x} v |^{2} d x d y + \frac{1 - s}{2} \int_{R^{2}} {| {(- Δ)}_{y}^{s / 2} v |}^{2} d x d y + \frac{1 + s}{2} \int_{R^{2}} | v |^{2} d x d y \\ = \frac{1 + s}{2_{s}} \int_{R^{2}} | v |^{2_{s}} d x d y + κ (1 + s) \int_{R^{2}} h (x, y) | u |^{α} | v |^{β} d x d y \\ + κ \int_{R^{2}} (y h_{y} (x, y) + s x h_{x} (x, y)) | u |^{α} | v |^{β} d x d y \\ + κ α \int_{R^{2}} (y h (x, y) | u |^{α - 2} u (\partial_{y} u) + s x h | u |^{α - 2} u \partial_{x} u) | v |^{β} d x d y . \end{aligned}$
(6.16)
Moreover, by multiplying first and second equation in (1.1) by u and v respectively, and integrating over $R^{2}$ we obtain the following identities:
$\begin{aligned} \int_{R^{2}} | \partial_{x} u |^{2} d x d y + \int_{R^{2}} {| {(- Δ)}_{y}^{s / 2} u |}^{2} d x d y + \int_{R^{2}} | u |^{2} d x d y = \int_{R^{2}} | u |^{2_{s}} d x d y + κ α \int_{R^{2}} h (x, y) | u |^{α} | v |^{β} d x d y \end{aligned}$
(6.17)
and
$\begin{aligned} \int_{R^{2}} | \partial_{x} v |^{2} d x d y + \int_{R^{2}} {| {(- Δ)}_{y}^{s / 2} v |}^{2} d x d y + \int_{R^{2}} | v |^{2} d x d y = \int_{R^{2}} | v |^{2_{s}} d x d y + κ β \int_{R^{2}} h (x, y) | u |^{α} | v |^{β} d x d y . \end{aligned}$
(6.18)
From (6.17) and (6.15), we deduce that
$\begin{aligned} s \int_{R^{2}} | \partial_{x} u |^{2} d x d y + s \int_{R^{2}} {| {(- Δ)}_{y}^{s / 2} u |}^{2} d x d y \\ = (1 + s) (\frac{1}{2} - \frac{1}{2_{s}}) \int_{R^{2}} | u |^{2_{s}} d x d y + \frac{κ (1 + s) (α - 2)}{2} \int_{R^{2}} h (x, y) | u |^{α} | v |^{β} d x d y \\ - κ \int_{R^{2}} (y h_{y} (x, y) + s x h_{x} (x, y)) | u |^{α} | v |^{β} d x d y \\ - κ β \int_{R^{2}} (y h (x, y) | v |^{β - 2} v (\partial_{y} v) + s x h | v |^{β - 2} v \partial_{x} v) | u |^{α} d x d y . \end{aligned}$
(6.19)
Further, from (6.18) and (6.16), we also obtain the following
$\begin{aligned} s \int_{R^{2}} | \partial_{x} v |^{2} d x d y + s \int_{R^{2}} {| {(- Δ)}_{y}^{s / 2} v |}^{2} d x d y \\ = (1 + s) (\frac{1}{2} - \frac{1}{2_{s}}) \int_{R^{2}} | v |^{2_{s}} d x d y + \frac{κ (1 + s) (β - 2)}{2} \int_{R^{2}} h (x, y) | u |^{α} | v |^{β} d x d y \\ - κ \int_{R^{2}} (y h_{y} (x, y) + s x h_{x} (x, y)) | u |^{α} | v |^{β} d x d y \\ - κ α \int_{R^{2}} (y h (x, y) | u |^{α - 2} u (\partial_{y} v) + s x h | u |^{α - 2} u \partial_{x} u) | v |^{β} d x d y . \end{aligned}$
(6.20)
Adding (6.19) and (6.20) gives
$\begin{aligned} s (\int_{R^{2}} | \partial_{x} u |^{2} d x d y + \int_{R^{2}} | \partial_{x} v |^{2} d x d y + \int_{R^{2}} {| {(- Δ)}_{y}^{s / 2} u |}^{2} d x d y + \int_{R^{2}} {| {(- Δ)}_{y}^{s / 2} v |}^{2} d x d y \\ - \int_{R^{2}} | u |^{2_{s}} d x d y - \int_{R^{2}} | v |^{2_{s}} d x d y) \\ = \frac{κ (1 + s) (2_{s} - 4)}{2} \int_{R^{2}} h (x, y) | u |^{α} | v |^{β} d x d y \\ - 2 κ \int_{R^{2}} (y h_{y} (x, y) + s x h_{x} (x, y)) | u |^{α} | v |^{β} d x d y \\ - κ \int_{R^{2}} h (x, y) | u |^{α} [y \partial_{y} (| v |^{β}) + s x \partial_{x} (| v |^{β})] d x d y - κ \int_{R^{2}} h (x, y) [y \partial_{y} (| u |^{α}) \\ + s x \partial_{x} (| u |^{α})] | v |^{β} d x d y \end{aligned}$
after simplifying the above expression, we get
$\begin{aligned} s (\int_{R^{2}} | \partial_{x} u |^{2} d x d y + \int_{R^{2}} | \partial_{x} v |^{2} d x d y + \int_{R^{2}} {| {(- Δ)}_{y}^{s / 2} u |}^{2} d x d y + \int_{R^{2}} {| {(- Δ)}_{y}^{s / 2} v |}^{2} d x d y \\ - \int_{R^{2}} | u |^{2_{s}} d x d y - \int_{R^{2}} | v |^{2_{s}} d x d y) \\ = κ s 2_{s} \int_{R^{2}} h (x, y) | u |^{α} | v |^{β} d x d y - κ \int_{R^{2}} (y h_{y} (x, y) + s x h_{x} (x, y)) | u |^{α} | v |^{β} d x d y . \end{aligned}$
(6.21)
Hence, the proof is complete.

Now we are going to prove the non-existence result using the previous lemma.
Proof of Theorem 8. Proof of Theorem 8.

$(a)$ Let $(u, v)$ be a solution to (1.1) such that $x u, y u \in L^{2} (R^{2})$ . We first divide the equation (6.1) by s and subsequently, we subtract (6.1) into (6.2) to get

\begin{aligned} ‖ u ‖_{2}^{2} + ‖ v ‖_{2}^{2} = \frac{κ}{s} \int_{R^{2}} (y h_{y} (x, y) + s x h_{x} (x, y)) | u |^{α} | v |^{β} d x d y . \end{aligned}

(6.22)

One can notice that

u = v = 0

for

κ = 0

i.e., there is only a trivial solution for

κ = 0

. Moreover, if h is a constant function and κ is any real number, from (6.22) we have

\begin{aligned} ‖ u ‖_{2}^{2} + ‖ v ‖_{2}^{2} = 0, \end{aligned}

which implies u, v are both identically zero a.e. Hence, there does not exist any non-trivial solution to (1.1) for this case.

$(b)$ We are given that $h \in L^{\infty} (R^{2}) \cap C^{1} (R^{2})$ satisfies $κ x h_{x} ⩽ 0$ , $κ y h_{y} ⩽ 0$ . Using the hypotheses on h in (6.22), we deduce that

\begin{aligned} ‖ u ‖_{2}^{2} + ‖ v ‖_{2}^{2} ⩽ 0. \end{aligned}

From above inequality, we can conclude that u, v are both identically zero almost everywhere. Hence, there does not exist any non-trivial solution to (1.1) for such h.

Example 26.

The following examples of h ensure the non-existence of solutions to (1.1). (a)

If we consider

\begin{aligned} h (x, y) = {\begin{cases} e^{- \frac{1}{1 - x^{2} - y^{2}}} & if \sqrt{x^{2} + y^{2}} < 1 \\ 0 & if \sqrt{x^{2} + y^{2}} ⩾ 1 \end{cases} \end{aligned}

then

h \in C_{c}^{\infty} (R^{2})

and

κ x h_{x} ⩽ 0

κ y h_{y} ⩽ 0

for

κ > 0

. Thus by Theorem (8), there does not exist any non-trivial solution

(u, v)

to (1.1) such that

x u, y u \in L^{2} (R^{2})

(b)

Moreover, if $h (x, y) = e^{\frac{1}{1 + x^{2} + y^{2}}}$ , then $h \in L^{\infty} (R^{2}) \cap C^{\infty} (R^{2})$ with $κ x h_{x} ⩽ 0$ and $κ y h_{y} ⩽ 0$ for $κ > 0$ . As a result, no non-trivial solution $(u, v)$ exists to (1.1) such that $x u, y u \in L^{2} (R^{2})$ .

Footnotes

Acknowledgements

RK acknowledges the support of the CSIR fellowship, file no. 09/1125(0016)/2020–EMR–I. TM acknowledges the support of the Start up Research Grant from DST-SERB, sanction no. SRG/2022/000524.

References

Bonder

J.F.

Saintier

Silva

, The concentration-compactness principle for fractional order Sobolev spaces in unbounded domains and applications to the generalized fractional Brézis-Nirenberg problem, NoDEA Nonlinear Differential Equations Appl. 25(6) (2018), 52, 25.

Brézis

Lieb

, A relation between pointwise convergence of functions and convergence of functionals, Proceedings of the American Mathematical Society 88(3) (1983), 486–490. doi:10.1090/S0002-9939-1983-0699419-3.

Di Nezza

Palatucci

Valdinoci

, Hitchhiker’s guide to the fractional Sobolev spaces, Bull. Sci. Math. 136(5) (2012), 521–573. doi:10.1016/j.bulsci.2011.12.004.

Esfahani

, Anisotropic Gagliardo–Nirenberg inequality with fractional derivatives, Z. Angew. Math. Phys. 66(6) (2015), 3345–3356. doi:10.1007/s00033-015-0586-y.

Esfahani

Ehsan Esfahani

, Positive and nodal solutions of the generalized BO–ZK equation, Rev. R. Acad. Cienc. Exactas Fís. Nat. Ser. A Mat. RACSAM 112(4) (2018), 1381–1390. doi:10.1007/s13398-017-0435-2.

Felmer

Wang

, Qualitative properties of positive solutions for mixed integro-differential equations, Discrete Contin. Dyn. Syst. 39(1) (2019), 369–393. doi:10.3934/dcds.2019015.

Gou

Hajaiej

Stefanov

A.G.

, On the solitary waves for anisotropic nonlinear Schrödinger models on the plane, Eur. J. Math. 9(3) (2023), 55, 34.

Lions

P.-L.

, The concentration-compactness principle in the calculus of variations. The limit case. I & II, Rev. Mat. Iberoamericana 1(1) (1985), 45–121, 145–201. doi:10.4171/rmi/6.

Xiang

Zhang

, A nonhomogeneous fractional p-Kirchhoff type problem involving critical exponent in

R^{N}

, Adv. Nonlinear Stud. 17(3) (2017), 611–640. doi:10.1515/ans-2016-6002.