Ground state solutions for the Hamilton–Choquard elliptic system with critical exponential growth

Abstract

In this paper, we study the following Hamilton–Choquard type elliptic system: $\begin{matrix} \{\begin{array}{ll} - Δ u + u = (I_{α} * F (v)) f (v), & x \in R^{2}, \\ - Δ v + v = (I_{β} * F (u)) f (u), & x \in R^{2}, \end{array} \end{matrix}$ where $I_{α}$ and $I_{β}$ are Riesz potentials, $f : R \to R$ possessing critical exponential growth at infinity and $F (t) = \int_{0}^{t} f (s) d s$ . Without the classic Ambrosetti–Rabinowitz condition and strictly monotonic condition on f, we will investigate the existence of ground state solution for the above system. The strongly indefinite characteristic of the system, combined with the convolution terms and critical exponential growth, makes such problem interesting and challenging to study. With the help of a proper auxiliary system, we employ an approximation scheme and the non-Nehari manifold method to control the minimax levels by a fine threshold, and succeed in restoring the compactness for the critical problem. Existence of a ground state solution is finally established by the concentration compactness argument and some detailed estimates.

Keywords

Hamilton–Choquard elliptic system Critical exponential growth Ground state solution Trudinger–Moser inequality

1. Introduction

In this paper, we mainly pay our attention to the existence of ground state solution for the following Hamilton–Choquard system, $\begin{matrix} (1.1) & \{\begin{array}{ll} - Δ u + u = (I_{α} * F (v)) f (v), & x \in R^{2}, \\ - Δ v + v = (I_{β} * F (u)) f (u), & x \in R^{2}, \end{array} \end{matrix}$ where $F (t) : = \int_{0}^{t} f (s) d s$ , $0 < α ⩽ β < 2$ , $I_{α}$ and $I_{β}$ are Riesz potentials defined by $\begin{aligned} I_{α} (x) = \frac{A_{α}}{| x |^{2 - α}}, A_{α} = \frac{Γ (\frac{2 - α}{2})}{2^{α} π Γ (\frac{α}{2})}, x \in R^{2} ∖ {0}; \\ I_{β} (x) = \frac{A_{β}}{| x |^{2 - β}}, A_{β} = \frac{Γ (\frac{2 - β}{2})}{2^{β} π Γ (\frac{β}{2})}, x \in R^{2} ∖ {0} . \end{aligned}$ The nonlinearity $f : R \to R$ satisfying following basic assumptions:

$f \in C (R, R)$ , and there exists $β_{0} > 0$ such that $\begin{matrix} (1.2) & lim_{| t | \to + \infty} \frac{f (t)}{e^{\tilde{β} t^{2}}} = 0, for all \tilde{β} > β_{0}, \end{matrix}$ and $\begin{matrix} (1.3) & lim_{| t | \to + \infty} \frac{f (t)}{e^{\tilde{β} t^{2}}} = + \infty, for all \tilde{β} < β_{0}; \end{matrix}$

$f (t) = 0$ if $t ⩽ 0$ and $f (t) = o (t)$ as $t \to 0^{+}$ ;

there exist $M_{0} > 0$ and $t_{0} > 0$ such that $\begin{matrix} 0 < F (t) ⩽ M_{0} f (t), \forall | t | ⩾ t_{0}; \end{matrix}$

there exist $p > 2$ and $K > 0$ such that $\begin{matrix} F (t) ⩾ K^{1 / 2} t^{p}, \forall t ⩾ 0, \end{matrix}$ where K is defined by (3.10);

$t \mapsto \frac{f (t)}{t}$ is nondecreasing on $t \in (0, + \infty)$ .

Setting $α = β$ and $u = v$ , we deduce the following Choquard equation from (1.1) which has attracted lots of researchers’ attentions in recent years, $\begin{matrix} (1.4) & \{\begin{array}{l} - Δ u + V (x) u = (W * F (u)) f (u), x \in R^{N}, \\ u \in H^{1} (R^{N}) . \end{array} \end{matrix}$ When $V \equiv 1$ , $W = \frac{1}{| x |}$ and $f (u) = u$ , the following classical Choquard equation was first proposed by Fröhlich [20] and Pekar [35] to model quantum polarons, $\begin{matrix} (1.5) & - Δ u + u = (\frac{1}{| x |} * | u |^{2}) u, x \in R^{3} . \end{matrix}$ In the context of a certain approximation to Hartree–Fock theory of one component plasma, (1.5) is used to describe the phenomenon of an electron trapped in its own hole. Existence and uniqueness solution to (1.5) were established by Lieb [25] via symmetric decreasing rearrangement inequalities. For case that the positive potential V is positive or sign-changing and periodic, Ackermann [1] investigated (1.4) under the following Ambrosetti–Rabinowitz type condition (AR)′, and demonstrated the existence of infinitely many solutions via an abstract critical point theorem.

There exists $θ > 2$ such that $2 t f (t) ⩾ θ F (t) > 0$ for all $t \in R ∖ {0}$ .

Later, these results were generalized and improved by Qin et al. [36] under some general assumptions on W and f. Equation (1.4) with Riesz potential (i.e.,

W = I_{α}

) has been widely investigated in recent years, and various results were obtained, such as, existence and multiplicity of solutions [3,23,27,32,33,42], sign-changing solutions [21,44], semiclassical solutions [31,45], etc.

Note that all results obtained above concern the case $N ⩾ 3$ and the nonlinearity f has polynomial growth at infinity. When $N = 2$ , f is allowed to has critical exponential growth at infinity with the aid of the following Trudinger–Moser inequality established by Cao [9], see also [2,10].

Lemma 1.1.
i) If $\tilde{β} > 0$ and $u \in H^{1} (R^{2})$ , then $\begin{matrix} \int_{R^{2}} (e^{\tilde{β} u^{2}} - 1) d x < \infty; \end{matrix}$ ii) if $u \in H^{1} (R^{2})$ , $‖ \nabla u ‖_{2}^{2} ⩽ 1$ , $‖ u ‖_{2} ⩽ M < \infty$ , and $\tilde{β} < 4 π$ , then there exists a constant $C (M, \tilde{β})$ , which depends only on M and $\tilde{β}$ , such that $\begin{matrix} \int_{R^{2}} (e^{\tilde{β} u^{2}} - 1) d x ⩽ C (M, \tilde{β}) . \end{matrix}$

In recent work [37], Qin and Tang studied (1.4) with Riesz potential and proved the existence of nontrivial solutions for case that V is periodic and 0 lies in a gap of the spectrum of the Schrödinger operator $- Δ + V$ . We emphasize that an approaching argument and a direct method were employed there (1.4), and the following inequalities play an important role in their proofs. For similar problem with doubly critical exponents, we refer to [12] and to [5,6,11,15] for related works.
Lemma 1.2 (Cauchy–Schwarz type inequality [30]).

For any $g, h \in L_{loc}^{1} (R^{2})$ , the following inequality holds, $\begin{matrix} (1.6) & \int_{R^{2}} (I_{α} * | g |) | h | d x ⩽ {(\int_{R^{2}} (I_{α} * | g |) | g | d x)}^{\frac{1}{2}} {(\int_{R^{2}} (I_{α} * | h |) | h | d x)}^{\frac{1}{2}} . \end{matrix}$

Lemma 1.3 (Hardy–Littlewood–Sobolev inequality [26]).

Let $s, r > 1$ and $0 < μ < 2$ with $\frac{1}{s} + \frac{1}{r} = \frac{4 - μ}{2}$ , $g \in L^{s} (R^{2})$ and $h \in L^{r} (R^{2})$ . There exists a sharp constant $C (μ, s, r)$ , independent of g, h, such that $\begin{matrix} (1.7) & \int_{R^{2}} (I_{2 - μ} * g) h d x ⩽ C (μ, s, r) ‖ g ‖_{s} ‖ h ‖_{r} . \end{matrix}$ In particular, $\begin{matrix} (1.8) & \int_{R^{2}} (I_{α} * g) h d x ⩽ C_{0} ‖ g ‖_{4 / (2 + α)} ‖ h ‖_{4 / (2 + α)}, \end{matrix}$ where $C_{0} : = C (2 - α, 4 / (2 + α), 4 / (2 + α))$ .

In recent decades, there has been increasing attention to the following Hamiltonian elliptic system, $\begin{matrix} (1.9) & \{\begin{array}{ll} - Δ u + V (x) u = f_{1} (x, v) & x \in Ω, \\ - Δ v + V (x) v = f_{2} (x, u) & x \in Ω, \end{array} \end{matrix}$ where $Ω \subset R^{2}$ is a domain, $V \in C (Ω, (0, \infty))$ and $f_{i} \in C (Ω \times R, R)$ , $i = 1, 2$ . When $V = 0$ and Ω is a bounded domain with a smooth boundary, de Figueiredo, do Ó and Ruf [13] proved the existence of solutions for system (1.9) under (F1)–(F3), the following classic Ambrosetti–Rabinowitz condition (AR) and other condition involving the behavior of the nonlinearity $f_{i}$ at infinity.

there exists $θ > 2$ such that $f_{i} (t) t ⩾ θ F_{i} (t) > 0$ for all $t \neq 0$ and $i = 1, 2$ .

Later, Lam and Lu [22] weakened the condition (AR) to following (L1), and showed the existence of nontrivial solutions for system (1.9) for both cases that f has subcritical and critical exponent growth.

there exists $θ > 2$ and $t_{1} > 0$ such that $f_{1} (t) t ⩾ θ F_{1} (t) > 0$ and $f_{2} (t) t ⩾ 2 F_{2} (t) > 0$ for all $t ⩾ t_{1}$ .

When

V \equiv 1

and

Ω = R^{2}

, de Figueiredo, do Ó and Zhang [14] studied (1.9) under (F1)–(F3), (AR), the following strictly monotonic condition (F5′) and technical condition (F6).

$t \mapsto \frac{f_{i} (t)}{| t |}$ is strictly increasing on $(0, \infty)$ for $i = 1, 2$ ;

${lim inf}_{| t | \to \infty} \frac{t f_{i} (t)}{e^{β_{0} t^{2}}} ⩾ κ_{0} > \frac{2 e}{α_{0}}$ , $i = 1, 2$ .

Existence of ground state solution was showed by them by using the generalized Nehari manifold method (cf. [40]). These results were improved recently by Qin et al. [38] by weakening (F5′) to (F5) and using the following general conditions instead of (AR) and (F6).

$f_{i} (x, t) t ⩾ 2 F_{i} (x, t) ⩾ 0$ for all $(x, t) \in R^{2} \times R$ , and $\begin{matrix} \frac{f_{i} (x, t)}{t} ⩾ \frac{1}{2 γ_{2}^{2}} \Rightarrow f_{i} (x, t) t - 2 F_{i} (x, t) > 0, i = 1, 2; \end{matrix}$

${lim inf}_{| t | \to \infty} \frac{t^{2} F_{i} (x, t)}{e^{β_{0} t^{2}}} ⩾ κ_{1} > \frac{V_{M}}{α_{0}^{2}}$ uniformly on $x \in R^{2}$ , where $V_{M} : = {max}_{R^{2}} V$ .

For related works with various assumptions on V to guarantee the compactness result, we draw the readers’ attention to papers [4,8,16,24] and to [39] for the periodic potential case.

In recent paper [28], Maia and Miyagaki studied the following Hamilton–Choquard type elliptic system, $\begin{matrix} (1.10) & \{\begin{array}{ll} - Δ u + V (x) u = (I_{α} * G (v)) g (v), & x \in R^{2}, \\ - Δ v + V (x) v = (I_{α} * F (u)) f (u), & x \in R^{2} . \end{array} \end{matrix}$ They [28] imposed some assumptions on V guaranteeing that the following embedding is compact for $p ⩾ 2$ , $\begin{matrix} (1.11) & H_{V}^{1} : = {u \in H^{1} (R^{2}) : \int_{R^{2}} V (x) u^{2} d x < \infty} ↪ L^{p} (R^{2}), \end{matrix}$ and found a positive solution of (1.10) by using a Galerkin approximation procedure provided that f, g satisfy (F1)–(F3), (AR), (F5′) and the following condition.

there exist constants $p > 2$ and $C_{p} > 0$ such that for all $s ⩾ 0$ , $\begin{matrix} F (s), G (s) ⩾ 2 {(\frac{C_{p}}{p})}^{\frac{1}{2}} s^{p}, \end{matrix}$ with $C_{p} > {(\frac{\bar{C} β_{0} (p - 1)}{4 π p})}^{\frac{p - 1}{p}} S_{p}^{2}$ , where $\bar{C} > \frac{4}{2 + α}$ , $0 < α < 2$ and $\begin{matrix} S_{p} = inf_{u \in H_{V}^{1}} \int_{R^{2}} (| \nabla u |^{2} + V (x) u^{2}) and \int_{R^{2}} (I_{α} * | u |^{p}) | u |^{p} = 1 . \end{matrix}$

Clearly, the compact embedding (1.11) plays an important role in their argument [28]. Later, Maia and Miyagaki [29] continued to consider the following system,

\begin{matrix} (1.12) & \{\begin{array}{ll} - Δ u + u = (I_{α} * | v |^{\frac{α}{2} + 1}) | v |^{\frac{α}{2} - 1} v + g (v) & x \in R^{2}, \\ - Δ v + v = (I_{β} * | u |^{\frac{β}{2} + 1}) | u |^{\frac{β}{2} - 1} u + f (u) & x \in R^{2} . \end{array} \end{matrix}

They obtained a ground state solution for (1.12) via the generalized Nehari manifold method when f, g satisfy (F1)–(F3), (AR) with

θ \in (2, min {2 + α, 2 + β}]

, (F5′) and the following condition.

there exist constants $2 < p < 4$ and $C_{p} > 0$ such that for all $s ⩾ 0$ , $\begin{matrix} F (s), G (s) ⩾ C_{p} s^{p}, \end{matrix}$ where $C_{p}$ is defined by [29, (5.3)].

We emphasize that the strictly monotonic condition (F5′) is crucial in applying the generalized Nehari manifold method (cf. [40]), it will become much more complicated to seek a ground state solution when one reduces (F5′) to (F5).

Recently, Deng and Yu [17] investigated the following Hamiltonian Choquard type elliptic systems involving singular weights, $\begin{matrix} (1.13) & \{\begin{array}{ll} - Δ u + V (x) u = (I_{α} * \frac{G (v)}{| x |^{α_{1}}}) \frac{g (v)}{| x |^{α_{1}}}, & x \in R^{2}, \\ - Δ v + V (x) v = (I_{β} * \frac{F (u)}{| x |^{β_{1}}}) \frac{f (u)}{| x |^{β_{1}}}, & x \in R^{2}, \end{array} \end{matrix}$ where $0 < α_{1} ⩽ \frac{α}{2}$ and $0 < β_{1} ⩽ \frac{β}{2}$ . They assumed some coercive condition on V such that (1.11) holds with $p ⩾ 1$ , and showed that (1.13) has a positive solution with $α = β$ and $α_{1} = β_{1}$ by using the Moser functions and linking theorem. Besides (F1)–(F3) and condition (AR), following condition (F6″) was introduced there [17].

${lim inf}_{s \to + \infty} \frac{s F (s)}{e^{β_{0} s^{2}}} ⩾ κ$ and ${lim inf}_{s \to + \infty} \frac{s G (s)}{e^{β_{0} s^{2}}} ⩾ κ$ with $\begin{matrix} κ > inf_{ρ > 0} \sqrt{\frac{(2 - 2 α_{1} + α) α (1 + α) (2 + α)}{8 π A_{α} ρ^{2 - 2 α_{1} + α} β_{0}^{2}} e^{\frac{V_{ρ} ρ^{2} (2 - 2 α_{1} + α)}{4} - 1}}, \end{matrix}$ and $V_{ρ} : = {sup}_{| x | ⩽ ρ} V (x)$ .

Similar result for fractional Hamiltonian Choquard type elliptic systems can be found in recent paper [18] via a fractional Trudinger–Moser inequality.

Motivated by above mentioned works, we will study system (1.1) in this paper and aim to find a ground state solution without the classic Ambrosetti–Rabinowitz condition (AR) and strictly monotonic condition (F5′). Thus the generalized Nehari manifold methods used in [14,29] and the argument employed by Maia and Miyagaki [28] are invalid in such case. Since the nonlinearity f possessing critical exponential growth at infinity (i.e., f satisfies (F1)), the Fréchet derivative of functional Φ (defined later by (2.4)) is no longer weakly sequentially continuous, so the generalized Linking theorem [19] is not applicable. Compared with the Hamiltonian elliptic (1.9), it is much more difficult to estimate the minimax level for (1.1) due to the presence of the convolution terms. To conquer the difficulties mentioned above, in the present paper, we shall employ an approximating argument and the non-Nehari manifold method to find some proper Cerami sequences at which the corresponding functional of (1.1) has bounded minimax levels. By investigating a proper auxiliary system, we find a fine threshold for the minimax levels and succeed in certifying the boundedness of the Cerami sequences. Then we show that the sequences is nonvanishing by using the concentration compactness argument, with the help of which we are allowed to establish the existence of ground state solution for (1.1) under the conditions (F1)–(F5).

Now we are ready to state our main result.

Theorem 1.4.
Let (F1)–(F5) be satisfied. Then system ( 1.1 ) has a ground state solution $\hat{z} \in H^{1} (R^{2}) \times H^{1} (R^{2})$ such that $\begin{array}{r} Φ (\hat{z}) = inf_{M} Φ > 0, \end{array}$ where $M$ is the Nehari–Pankov manifold defined later by ( 2.8 ).

Since the working space is $R^{2}$ and the nonlinearity f has critical exponential growth, we must deal with the compactness issue. With the help of technical condition (F4), we can compare the relationship of energy levels between system (1.1) and a proper auxiliary system where the nonlinearity has polynomial growth at infinity. The role it plays is the same as (F4′) which helps us to find a fine threshold for minimax levels.
Remark 1.5.
Note that (F5) is weaker than (F5′). Conditions like (F6), (F6′) and (F6″) involving the behavior of the nonlinearity f at infinity are also used to control the minimax levels by a fine threshold in order to restore the compactness. However, condition (F6″) is not sufficient to get the estimate [17, (6.3)] in their argument, see the proof of Case (ii) in [17, Lemma 6.1]. Thus, it is very interesting to seek a ground state solution of (1.1) under such similar condition. We do not know whether Theorem 1.4 still holds or not if the condition (F5) is weakened to following general version.
$t \mapsto f (t)$ is nondecreasing on $t \in (0, + \infty)$ .

This paper is organized as follows. In Section 2, we introduce the variational setting and give some preliminary results. In Section 3, we investigate an proper auxiliary system and estimate the threshold value of Φ. Section 4 is dedicated to the proof of Theorem 1.4.
2. Variational framework and preliminaries

In this section, we give the variational framework and introduce some useful lemmas. In the Hilbert space $H^{1} (R^{2})$ , we use following inner product and norm, $\begin{matrix} (2.1) & (u, v) : = \int_{R^{2}} [\nabla u \nabla v + u v] d x, ‖ u ‖^{2} : = (u, u), \forall u, v \in H^{1} (R^{2}) . \end{matrix}$ By the Sobolev embedding theorem, there exists $γ_{s} \in (0, \infty)$ such that for any $s \in [2, + \infty)$ , $\begin{matrix} (2.2) & ‖ u ‖_{L^{s}} ⩽ γ_{s} ‖ u ‖, \forall u \in H^{1} (R^{2}) . \end{matrix}$ Let $E : = H^{1} (R^{2}) \times H^{1} (R^{2})$ and define inner product in E as follows, $\begin{matrix} (z_{1}, z_{2}) : = (u_{1}, u_{2}) + (v_{1}, v_{2}), \forall z_{i} = (u_{i}, v_{i}) \in E, i = 1, 2 . \end{matrix}$ Clearly, E is a Hilbert space, and the corresponding norm can defined as $\begin{matrix} ‖ z ‖ : = \sqrt{‖ u ‖^{2} + ‖ v ‖^{2}}, \forall z = (u, v) \in E . \end{matrix}$ Define subspaces of E as follows, $\begin{matrix} E^{+} : = {(u, u) : u \in H^{1} (R^{2})} and E^{-} : = {(u, - u) : u \in H^{1} (R^{2})} . \end{matrix}$ It is easy to check that $E^{+}$ is orthogonal to $E^{-}$ with respect to the inner product $(\cdot, \cdot)$ . For each $z = (u, v) \in E$ , we define $\begin{matrix} z^{+} : = (\frac{u + v}{2}, \frac{u + v}{2}) and z^{-} : = (\frac{u - v}{2}, \frac{v - u}{2}), \end{matrix}$ by noting that $z^{+} \in E^{+}$ and $z^{-} \in E^{-}$ , we have $E = E^{+} \oplus E^{-}$ .

For any $ε > 0$ , $\tilde{β} > β_{0}$ and $q > 0$ , it follows from (F1) and (F2) that there exists $C_{ε} = C (ε, \tilde{β}) > 0$ such that $\begin{matrix} (2.3) & | F (t) | ⩽ ε t^{2} + C_{ε} | t | (e^{\tilde{β} t^{2}} - 1), \forall (x, t) \in R^{2} \times R . \end{matrix}$ By virtue of (F1), (2.3), Lemma 1.1 and Lemma 1.3, we can define the energy functional corresponding to the system (1.1), $\begin{matrix} (2.4) & Φ (u, v) = \int_{R^{2}} (\nabla u \cdot \nabla v + u v) d x - \frac{1}{2} \int_{R^{2}} [(I_{α} * F (v)) F (v) + (I_{β} * F (u) F (u))] d x . \end{matrix}$ It is not difficult to verify that Φ is well defined on E and $Φ \in C^{1} (E, R)$ , moreover, for any $z = (u, v)$ , $ζ = (ϕ, ψ) \in E$ , there holds $\begin{array}{rcl} ⟨ Φ^{'} (z), ζ ⟩ & = & \int_{R^{2}} [\nabla u \cdot \nabla ψ + \nabla ϕ \cdot \nabla v + (u ψ + ϕ v)] d x \\ (2.5) & - \int_{R^{2}} [(I_{α} * F (v)) f (v) ψ + (I_{β} * F (u) f (u)) ϕ] d x . \end{array}$ Then, for any $z = (u, v) \in E$ , we have $\begin{array}{r} (2.6) & Φ (z) = \frac{1}{2} ({‖ z^{+} ‖}^{2} - {‖ z^{-} ‖}^{2}) - \frac{1}{2} \int_{R^{2}} [(I_{α} * F (v)) F (v) + (I_{β} * F (u) F (u))] d x, \end{array}$ and $\begin{array}{rcl} ⟨ Φ^{'} (z), z ⟩ & = & {‖ z^{+} ‖}^{2} - {‖ z^{-} ‖}^{2} - \int_{R^{2}} [(I_{α} * F (v)) f (v) v + (I_{β} * F (u) f (u)) u] d x \\ (2.7) & = & 2 \int_{R^{2}} [\nabla u \nabla v + u v] d x - \int_{R^{2}} [(I_{α} * F (v)) f (v) v + (I_{β} * F (u) f (u)) u] d x . \end{array}$ Note that solutions of (1.1) correspond to the critical points of Φ.

Define the Nehari–Pankov manifold introduced first by Pankov [34], $\begin{array}{r} (2.8) & M : = {z \in E ∖ E^{-} : ⟨ Φ^{'} (z), z ⟩ = ⟨ Φ^{'} (z), ζ ⟩ = 0, \forall ζ \in E^{-}} . \end{array}$ We shall find the ground state solution of system (1.1) by minimizing the functional Φ on such set.

Lemma 2.1 ([1, Lemma 2.4]).

The following inequality holds, $\begin{matrix} s t ⩽ \{\begin{array}{ll} (e^{t^{2}} - 1) + s {(log s)}^{\frac{1}{2}}, & t ⩾ 0, s ⩾ e^{\frac{1}{4}}; \\ (e^{t^{2}} - 1) + \frac{1}{2} s^{2}, & t ⩾ 0, 0 ⩽ s ⩽ e^{\frac{1}{4}} . \end{array} \end{matrix}$

Lemma 2.2.
Let (F1) and (F2) be satisfied. There exists a constant $\bar{ρ} > 0$ such that $\begin{array}{r} (2.9) & κ_{0} : = inf {Φ (z) : z \in E^{+}, ‖ z ‖ = \bar{ρ}} > 0 . \end{array}$
Proof.
Note that $\begin{matrix} ‖ \nabla u ‖_{2} ⩽ ‖ u ‖, \forall u \in H^{1} (R^{2}) . \end{matrix}$ (F1) and (F2) yield the existence of $\tilde{β} > β_{0}$ and $C_{1} > 0$ such that $\begin{array}{r} (2.10) & | F (t) | ⩽ | t |^{\frac{(2 + α)}{2}} + C_{1} (e^{\tilde{β} t^{2}} - 1) | t |^{3}, \forall t \in R . \end{array}$ Using (2.10), the Hölder inequality, and Lemma 1.1-ii), we have $\begin{array}{rcl} \int_{R^{2}} {| F (v) |}^{\frac{4}{2 + α}} d x & ⩽ & \int_{R^{2}} {[| v |^{\frac{(2 + α)}{2}} + C_{1} (e^{\tilde{β} v^{2}} - 1) | v |^{3}]}^{\frac{4}{2 + α}} d x \\ ⩽ & 2 \int_{R^{2}} | v |^{2} + {C_{1}}^{\frac{4}{2 + α}} {(e^{\tilde{β} v^{2}} - 1)}^{\frac{4}{2 + α}} | v |^{\frac{12}{2 + α}} d x \\ ⩽ & 2 {‖ v ‖_{2}^{2} + {C_{1}}^{\frac{4}{2 + α}} {(\int_{R^{2}} {(e^{\tilde{β} v^{2}} - 1)}^{\frac{4}{α}} d x)}^{\frac{α}{2 + α}} ‖ v ‖_{6}^{\frac{12}{2 + α}}} \\ ⩽ & 2 {‖ v ‖_{2}^{2} + {C_{1}}^{\frac{4}{2 + α}} {(\int_{R^{2}} e^{4 \tilde{β} α^{- 1} ‖ v ‖^{2} {(v / ‖ v ‖)}^{2}} - 1 d x)}^{\frac{α}{2 + α}} ‖ v ‖_{6}^{\frac{12}{2 + α}}} \\ (2.11) & ⩽ & 2 (‖ v ‖_{2}^{2} + C_{2} ‖ v ‖_{6}^{\frac{12}{2 + α}}), \forall ‖ v ‖ ⩽ \sqrt{\frac{α π}{2 \tilde{β}}} . \end{array}$ It follows from (1.8) and (2.11) that $\begin{array}{rcl} \int_{R^{2}} (I_{α} * F (v)) F (v) d x & ⩽ & C_{0} {(\int_{R^{2}} {| F (v) |}^{\frac{4}{2 + α}} d x)}^{\frac{2 + α}{2}} \\ ⩽ & C_{0} {[2 ‖ v ‖_{2}^{2} + C_{2} ‖ v ‖_{6}^{\frac{12}{2 + α}}]}^{\frac{2 + α}{2}} \\ (2.12) & ⩽ & 4 C_{0} ‖ v ‖_{2}^{2 + α} + 2 {C_{2}}^{\frac{2 + α}{2}} C_{0} ‖ v ‖_{6}^{6}, \forall ‖ v ‖ ⩽ \sqrt{\frac{α π}{2 \tilde{β}}} . \end{array}$ Similarly, we have $\begin{array}{rcl} (2.13) & \int_{R^{2}} (I_{β} * F (u)) F (u) d x & ⩽ & 4 C_{0} ‖ u ‖_{2}^{2 + β} + 2 {C_{2}}^{\frac{2 + β}{2}} C_{0} ‖ u ‖_{6}^{6}, \forall ‖ u ‖ ⩽ \sqrt{\frac{β π}{2 \tilde{β}}} . \end{array}$ Then by $α ⩽ β$ , we have for any $z \in E^{+}$ with $‖ z ‖ ⩽ \sqrt{\frac{α π}{2 \tilde{β}}}$ , $\begin{array}{rcl} Φ (z) & = & \frac{1}{2} ‖ z ‖^{2} - \frac{1}{2} \int_{R^{2}} [(I_{α} * F (v)) F (v) + (I_{β} * F (u)) F (u)] d x \\ ⩾ & \frac{1}{2} ‖ z ‖^{2} - C_{3} (‖ v ‖^{2 + α} + ‖ u ‖^{2 + β} + ‖ u ‖^{6} + ‖ v ‖^{6}) \\ ⩾ & \frac{1}{2} ‖ z ‖^{2} - C_{3} (‖ z ‖^{2 + α} + ‖ z ‖^{2 + β} + 2 ‖ z ‖^{6}) . \end{array}$ From $α, β > 0$ , we deduce that there exists $0 < \bar{ρ} < \sqrt{α π / 2 \tilde{β}}$ such that (2.9) holds. □

By using a similar argument as [36, Lemma 3.3], we have the following lemma.
Lemma 2.3.
Let (F1), (F2), and (F3) be satisfied and set $(e, e) \in E^{+}$ with $‖ (e, e) ‖ = 1$ . Then, there exists a constant $r_{0} > \bar{ρ}$ such that $sup Φ (\partial Q) ⩽ 0$ , where Q is defined as follows, $\begin{matrix} (2.14) & Q = {(w, - w) + s (e, e) : (w, - w) \in E^{-}, 0 ⩽ s ⩽ r_{0}, ‖ w ‖ ⩽ r_{0}} . \end{matrix}$

Based on the monotonicity condition (F5), we have the following lemma.
Lemma 2.4.
Assume that (F1), (F2) and (F5) hold. Then $\begin{array}{r} (2.15) & Φ (z) ⩾ Φ (t z + ζ) + \frac{1}{2} ‖ ζ ‖^{2} + \frac{1 - t^{2}}{2} ⟨ Φ^{'} (z), z ⟩ - t ⟨ Φ^{'} (z), ζ ⟩, \forall z \in E, ζ \in E^{-}, t ⩾ 0 . \end{array}$
Proof.
Set $z = (u, v)$ and $ζ = (w, - w)$ , and define $\begin{array}{r} (2.16) & \tilde{f} (t) : = (I_{α} * F (t)) f (t), \tilde{g} (t) : = (I_{β} * F (t)) f (t), \end{array}$ and $\begin{array}{r} (2.17) & \tilde{F} (t) = \int_{0}^{t} \tilde{f} (s) d s, \tilde{G} (t) = \int_{0}^{t} \tilde{g} (s) d s . \end{array}$ Using [40, Lemma 2.2] and (F5), we have $\begin{array}{rcl} Φ (z) - Φ (t z + ζ) & = & \frac{1}{2} ‖ ζ ‖^{2} + (1 - t^{2}) ⟨ u, v ⟩ + t ⟨ u, w ⟩ - t ⟨ v, w ⟩ \\ - \frac{1}{2} \int_{R^{2}} [(I_{α} * F (v)) F (v) - (I_{α} * F (t v - w)) F (t v - w)] d x \\ - \frac{1}{2} \int_{R^{2}} [(I_{β} * F (u)) F (u) - (I_{β} * F (t u + w)) F (t u + w)] d x \\ = & \frac{1}{2} ‖ ζ ‖^{2} + \frac{(1 - t^{2})}{2} ⟨ Φ^{'} (z), z ⟩ - t ⟨ Φ^{'} (z), ζ ⟩ \\ - \int_{R^{2}} \tilde{f} (v) (\frac{t^{2} - 1}{2} v - t w) + [\tilde{F} (v) - \tilde{F} (t v - w)] d x \\ - \int_{R^{2}} \tilde{g} (u) (\frac{t^{2} - 1}{2} u + t w) + [\tilde{G} (u) - \tilde{G} (t u + w)] d x \\ (2.18) & ⩾ & \frac{1}{2} ‖ ζ ‖^{2} + \frac{(1 - t^{2})}{2} ⟨ Φ^{'} (z), z ⟩ - t ⟨ Φ^{'} (z), ζ ⟩ . \end{array}$ □

From Lemma 2.4 we deduce the following corollary.
Corollary 2.5.
Let (F1), (F2) and (F5) be satisfied. Then $\begin{array}{r} (2.19) & Φ (z) ⩾ max_{t ⩾ 0, ζ \in E^{-}} Φ (t z + ζ), \forall z \in M . \end{array}$

In view of Lemma 2.2 and 2.3, we can prove the following lemma similarly as [41, Lemma 2.11].
Lemma 2.6.
Let (F1), (F2) and (F5) be satisfied. Then for any $z \in E ∖ E^{-}$ , there exist $t_{z} > 0$ and $ζ_{z} \in E^{-}$ such that $t_{z} z + ζ_{z} \in M$ .

Combining Corollary 2.5 with Lemma 2.6, we get the following result. Corollary 2.7.
Let $(F 1), (F 2)$ and $(F 5)$ be satisfied. Then $\begin{array}{r} (2.20) & b : = inf_{M} Φ = inf_{z \in E ∖ E^{-}} max_{t ⩾ 0, ζ \in E^{-}} Φ (t z + ζ) > 0 . \end{array}$

3. Estimates of minimax level

In this section, we introduce an auxiliary system and estimate the threshold value of the energy functional by comparing the auxiliary system with problem (1.1).

Consider the following auxiliary system $\begin{matrix} (3.1) & \{\begin{array}{l} - Δ u + u = (I_{α} * | v |^{p}) | v |^{p - 2} v, x \in R^{2}, \\ - Δ v + v = (I_{β} * | u |^{p}) | u |^{p - 2} u, x \in R^{2}, \end{array} \end{matrix}$ where $p > 2$ . Define the corresponding energy functional as follows, $\begin{matrix} (3.2) & E (u, v) = \int_{R^{2}} (\nabla u \nabla v + u v) d x - \frac{1}{2 p} \int_{R^{2}} [(I_{β} * | u |^{p}) | u |^{p} + (I_{α} * | v |^{p}) | v |^{p}] d x . \end{matrix}$ Let $\begin{matrix} H (z) = (I_{β} * | u |^{p}) | u |^{p} + (I_{α} * | v |^{p}) | v |^{p}, z = (u, v) \in E, \end{matrix}$ then $\begin{matrix} E (z) = \frac{1}{2} {‖ z^{+} ‖}^{2} - \frac{1}{2} {‖ z^{-} ‖}^{2} - \frac{1}{2 p} \int_{R^{2}} H (z) d x . \end{matrix}$ Set $\frac{1}{R} z = \frac{1}{R} (u, v)$ , we have $\begin{matrix} (3.3) & H (\frac{1}{R} z) = \frac{1}{R^{2 p}} [(I_{β} * | u |^{p}) | u |^{p} + (I_{α} * | v |^{p}) | v |^{p}] = \frac{1}{R^{2 p}} H (z) . \end{matrix}$ For any $e \in E^{+}$ , we define $\begin{matrix} E (e) = E^{-} \oplus R e, \end{matrix}$ and $\begin{matrix} Q (e, R) = {z \in E (e) : z = t e + ζ, ζ \in E^{-}, t ⩾ 0, ‖ z ‖ ⩽ R}, S (r) = {z \in E : ‖ z ‖ = r} . \end{matrix}$

Lemma 3.1.
The functional $E$ has Linking structure.
${inf}_{S (r) \cap E^{+}} E ⩾ δ > 0$ ;

There exists $R = R (e) > 0$ , such that ${sup}_{\partial Q (e, R)} E ⩽ 0$ , for any $e \in E^{+} ∖ {0}$ .

Proof.
i) can be verified easily by Lemma 1.1-i). Next, we prove that ii) holds. Arguing indirectly, assume that for some $e \in E^{+} ∖ {0}$ , there exists $R_{n} \to \infty$ , $z_{n} = t_{n} e + ζ_{n} \in \partial Q (e, R_{n})$ , with $t_{n} ⩾ 0$ , $ζ_{n} \in E^{-}$ such that $E (z_{n}) > 0$ . Since $z_{n} = t_{n} e + ζ_{n} \in \partial Q (e, R_{n})$ , and $E (ζ) ⩽ 0$ for all $ζ \in E^{-}$ , we have $t_{n} > 0$ and $‖ z_{n} ‖ = R_{n}$ . Note that $\begin{matrix} 0 ⩽ E (z_{n}) ⩽ \frac{1}{2} {‖ z_{n}^{+} ‖}^{2} - \frac{1}{2} {‖ z_{n}^{-} ‖}^{2} . \end{matrix}$ Then $‖ z_{n}^{-} ‖^{2} ⩽ ‖ z_{n}^{+} ‖^{2}$ , which combines with $‖ z_{n}^{-} ‖^{2} + ‖ z_{n}^{+} ‖^{2} = ‖ z_{n} ‖^{2}$ implies that $\begin{matrix} {‖ z_{n}^{+} ‖}^{2} ⩾ \frac{1}{2} ‖ z_{n} ‖^{2} = \frac{1}{2} R_{n}^{2} . \end{matrix}$ Thus $\begin{matrix} R_{n}^{2} = ‖ z_{n} ‖^{2} ⩾ {‖ z_{n}^{+} ‖}^{2} = ‖ t_{n} e ‖^{2} ⩾ \frac{1}{2} R_{n}^{2}, \end{matrix}$ and so $t_{n}^{2} \in [\frac{R_{n}^{2}}{2 ‖ e ‖^{2}}, \frac{R_{n}^{2}}{‖ e ‖^{2}}]$ .

Define $\begin{matrix} {\tilde{z}}_{n} = \frac{1}{R_{n}} z_{n} = \frac{t_{n}}{R_{n}} e + \frac{1}{R_{n}} ζ_{n} . \end{matrix}$ By $t_{n}^{2} \in [\frac{R_{n}^{2}}{2 ‖ e ‖^{2}}, \frac{R_{n}^{2}}{‖ e ‖^{2}}]$ and $‖ ζ_{n} ‖ ⩽ ‖ z_{n} ‖ = R_{n}$ , up to a subsequence, we may assume that ${\tilde{z}}_{n} ⇀ t_{0} e + ζ_{0}$ , where $t_{0} \in [\frac{1}{\sqrt{2} ‖ e ‖}, \frac{1}{‖ e ‖}]$ and $ζ_{0} \in E^{-}$ . Let $z_{0} = t_{0} e + ζ_{0}$ , obviously, $z_{0} \neq 0$ . By the following inequality $\begin{array}{r} (3.4) & 0 < E (z_{n}) = \frac{1}{2} {‖ z_{n}^{+} ‖}^{2} - \frac{1}{2} {‖ z_{n}^{-} ‖}^{2} - \frac{1}{2 p} \int_{R^{2}} H (z_{n}) d x, \end{array}$ we have $\begin{matrix} (3.5) & \frac{1}{2 p} \int_{R^{2}} \frac{1}{| R_{n} |^{2 p}} H (z_{n}) d x ⩽ \frac{1}{2 | R_{n} |^{2 p}} {‖ z_{n}^{+} ‖}^{2} - \frac{1}{2 | R_{n} |^{2 p}} {‖ z_{n}^{-} ‖}^{2} ⩽ \frac{1}{2 | R_{n} |^{2 p - 2}} = o_{n} (1) . \end{matrix}$ From (3.3), we know that $\begin{matrix} \frac{1}{| R_{n} |^{2 p}} H (z_{n}) = H (\frac{1}{R_{n}} z_{n}) = H ({\tilde{z}}_{n}) . \end{matrix}$ Since ${\tilde{z}}_{n} ⇀ t_{0} e + ζ_{0} = z_{0}$ , it follows from Fatou’s lemma and (3.5) that $\begin{array}{rcl} 0 & ⩾ & \underset{n \to \infty}{lim inf} \int_{R^{2}} \frac{1}{| R_{n} |^{2 p}} H (z_{n}) d x ⩾ \int_{R^{2}} \underset{n \to \infty}{lim inf} \frac{1}{| R_{n} |^{2 p}} H (z_{n}) d x \\ = & \int_{R^{2}} \underset{n \to \infty}{lim inf} H ({\tilde{z}}_{n}) d x = \int_{R^{2}} H (z_{0}) d x, \end{array}$ which contradicts with $z_{0} \neq 0$ . □

Consider a new norm in E by setting $\begin{matrix} ‖ z ‖_{τ} : = ‖ z^{+} ‖ + \sum_{j = 1}^{\infty} \frac{1}{2^{j}} | ⟨ z^{-}, e_{j} ⟩ |, \end{matrix}$ where ${e_{j}}_{j = 1}^{\infty}$ is a total orthonormal sequence in $E^{-}$ . The topology generated by $‖ \cdot ‖_{τ}$ will be denoted by τ and all topological notions related to it will includes this symbol. Denote the $wea k^{}$ topology on $E^{}$ as $(E^{}, T_{ζ^{}})$ .
Lemma 3.2.
$E_{b} : = {z \in E, E (z) ⩾ b}$ is τ-closed, and $E^{'} : (E_{b}, T_{τ}) \to (E^{}, T_{ζ^{}})$ is continuous, for any $b \in R$ .
Proof.
Setting sequence ${z_{n}} \subset E_{b}$ , with $z_{n} \overset{τ}{\to} z$ in E, we have $z_{n}^{+} \to z^{+}$ strongly in E, then ${z_{n}^{+}}$ is bounded in E. This, together with $b ⩽ E (z_{n}) ⩽ \frac{1}{2} ‖ z_{n}^{+} ‖^{2} - \frac{1}{2} ‖ z_{n}^{-} ‖^{2}$ , yields that $‖ z_{n}^{-} ‖$ and $‖ z_{n} ‖$ are also bounded. Thus $z_{n}^{-} ⇀ z^{-}$ , $z_{n} ⇀ z$ in E, as $n \to \infty$ . Consequently, $\begin{matrix} \underset{n \to \infty}{lim inf} (\frac{1}{2} {‖ z_{n}^{-} ‖}^{2} + \frac{1}{2 p} \int_{R^{2}} H (z_{n}) d x) ⩾ \frac{1}{2} {‖ z^{-} ‖}^{2} + \frac{1}{2 p} \int_{R^{2}} H (z) d x . \end{matrix}$ It follow from $z_{n}^{+} \to z^{+}$ in E that $\begin{aligned} E (z) & ⩾ lim_{n \to \infty} (\frac{1}{2} {‖ z_{n}^{+} ‖}^{2}) + \underset{n \to \infty}{lim sup} (- \frac{1}{2} {‖ z_{n}^{-} ‖}^{2} - \frac{1}{2 p} \int_{R^{2}} H (z_{n}) d x) \\ ⩾ \underset{n \to \infty}{lim sup} (\frac{1}{2} {‖ z_{n}^{+} ‖}^{2} - \frac{1}{2} {‖ z_{n}^{-} ‖}^{2} - \frac{1}{2 p} \int_{R^{2}} H (z_{n}) d x) \\ = \underset{n \to \infty}{lim sup} E (z_{n}) ⩾ b . \end{aligned}$ Thus $z \in E_{b}$ and $E_{b}$ is τ-closed.

Next we proof $E^{'} : (E_{b}, T_{τ}) \to (E^{}, T_{ζ^{}})$ is continuous. Choose sequence ${z_{n}} \subset E_{b}$ such that $z_{n} \overset{τ}{\to} z$ in E, we have $z_{n} ⇀ z$ in E. Therefore, for any $φ \in C_{0}^{\infty} (R^{2})$ , we obtain $E^{'} (z_{n}) φ \to E^{'} (z) φ$ , as $n \to \infty$ . Hence, $E^{'} : (E_{b}, T_{τ}) \to (E^{}, T_{ζ^{}})$ is continuous. □

According to Lemma 3.1 and Lemma 3.2, by using the generalized Linking theorem in E (cf. [43, Theorem 6.10]), we can obtain the Palais–Smale sequence for the function $E$ at the level c, where $\begin{matrix} (3.6) & c \in [δ, sup_{u \in Q (e, R)} E (u)] . \end{matrix}$
Lemma 3.3.
There exists a solution $z \in E ∖ {0}$ of system ( 3.1 ), such that $E (z) ⩽ c$ .
Proof.
Let ${z_{n}} \in E$ be a Palais–Smale sequence for $E$ at level $c \in [δ, {sup}_{u \in E (e)} E (u)]$ , we claim that ${z_{n}}$ is bounded in E. Assume that $‖ z_{n} ‖ \to \infty$ , as $n \to \infty$ . In view of $p > 2$ and (3.2), we have $\begin{matrix} (3.7) & ⟨ E^{'} (z_{n}), z_{n} ⟩ = ‖ z_{n} ‖^{2} - \int_{R^{2}} H (z_{n}) d x, \end{matrix}$ and $\begin{matrix} (3.8) & c + o_{n} (1) ‖ z_{n} ‖ = | E (z_{n}) - \frac{1}{2} ⟨ E^{'} (z_{n}), z_{n} ⟩ | = (\frac{1}{2} - \frac{1}{2 p}) \int_{R^{2}} H (z_{n}) d x . \end{matrix}$ From (3.7) and (3.8), we deduce that $\begin{matrix} (3.9) & \frac{o_{n} (1)}{‖ z_{n} ‖} ⩾ \frac{⟨ E^{'} (z_{n}), z_{n} ⟩}{‖ z_{n} ‖^{2}} = 1 - \frac{2 p (c + o_{n} (1) ‖ z_{n} ‖)}{(p - 1) ‖ z_{n} ‖^{2}} = 1 - \frac{o_{n} (1)}{‖ z_{n} ‖}, \end{matrix}$ which is a contradiction, then we show that ${z_{n}}$ is bounded. Note that $\begin{matrix} \frac{1}{2} δ ⩽ E (z_{n}) ⩽ ‖ z_{n} ‖^{2} . \end{matrix}$ Suppose that $\begin{matrix} ν : = \underset{n \to \infty}{lim sup} sup_{x \in R^{2}} \int_{B (x, 1)} | z_{n} |^{2} d x = 0, \end{matrix}$ then by Lions’ concentration compactness principle, $z_{n} \to 0$ in $L^{q} (R^{2})$ , with $q \in (2, \infty)$ . By applying $E^{'} (z_{n})$ to $(z_{n}^{+} - z_{n}^{-})$ , and using (1.8), we have $\begin{matrix} ‖ z_{n} ‖^{2} ⩽ C ‖ z_{n} ‖_{\frac{4 p}{2 + α}}^{2 p} + o_{n} (1) ‖ z_{n} ‖ = o_{n} (1) + o_{n} (1) ‖ z_{n} ‖ = o_{n} (1), \end{matrix}$ this contradiction shows that $ν > 0$ . Going if necessary to a subsequence, we may assume that there exists ${p_{n}} \subset Z^{2}$ such that ${\tilde{z}}_{n} : = z_{n} (\cdot - p_{n}) ⇀ z \neq 0$ . Note that ${{\tilde{z}}_{n}}$ is ${(P S)}_{c}$ sequence in E. Thus $E^{'} (z) = 0$ .

Next, we claim $E (z) ⩽ c = {lim}_{n \to \infty} E ({\tilde{z}}_{n})$ . It follows from Fatou’s lemma that $\begin{aligned} E (z) = E (z) - \frac{1}{2} ⟨ E^{'} (z), z ⟩ & = (\frac{1}{2} - \frac{1}{2 p}) \int_{R^{2}} H (z) d x \\ ⩽ (\frac{1}{2} - \frac{1}{2 p}) \underset{n \to \infty}{lim inf} \int_{R^{2}} H ({\tilde{z}}_{n}) d x \\ = \underset{n \to \infty}{lim inf} (E ({\tilde{z}}_{n}) - \frac{1}{2} ⟨ E^{'} ({\tilde{z}}_{n}), {\tilde{z}}_{n} ⟩) \\ = c + o_{n} (1) . \end{aligned}$ We complete the proof. □

From Lemma 3.3, we deduce that $\begin{matrix} K : = {z \in E ∖ {0} : E^{'} (z) = 0} \neq \emptyset . \end{matrix}$ Define the Nehari–Pankov manifold for functional $E$ , $\begin{matrix} N : = {z \in E ∖ {0} : ⟨ E^{'} (z), z ⟩ = ⟨ E^{'} (z), η ⟩ = 0, \forall η \in E^{-}} . \end{matrix}$

In view of Corollary 2.5, Lemma 2.6 and Lemma 3.1, we have the following lemma.
Lemma 3.4.

$E (z) = {max}_{t ⩾ 0, η \in E^{-}} E (t z^{+} + η) ⩾ {inf}_{S (r) \cap E^{+}} E ⩾ δ$ , for every $z \in N$ ;

For any $e \in E^{+} ∖ {0}$ , there exist unique constant $t > 0$ and $η \in E^{-}$ such that $t e + η \in N$ .

Lemma 3.5.
Let $c_{1} : = {inf}_{z \in K} E (z)$ . Then $c_{1}$ is achieved and $c_{1} = {inf}_{z \in N} E (z)$ .
Proof.
For any $z \in K$ , we have $z \in N$ . It follows from Lemma 3.4 that $E (z) ⩾ {inf}_{S (r) \cap E^{+}} E ⩾ δ$ . Therefore $c_{1} ⩾ δ$ . Assume that ${z_{n}} \subset K$ is a minimizing sequence for $E$ at $c_{1}$ , then $E^{'} (z_{n}) = 0$ and $E (z_{n}) \to c_{1}$ as $n \to \infty$ . Using the same argument as in the proof of Lemma 3.3, there exists $z_{0} \in K \subset N$ such that $E (z_{0}) ⩽ c_{1}$ . Moreover, by the definition of $c_{1}$ , we can show that $E (z_{0}) = c_{1}$ and so $c_{1}$ is achieved at $z_{0}$ . Obviously, $c_{1} ⩾ {inf}_{z \in N} E (z)$ . If $c_{1} > {inf}_{z \in N} E (z)$ , there exists $\bar{z} \in N$ such that $E (\bar{z}) < c_{1}$ . Set $\bar{z} = e + ξ$ , $e \in E^{+}$ and $ξ \in E^{-}$ . Using Lemma 3.4, we have $\begin{matrix} c_{1} > E (\bar{z}) = max_{t ⩾ 0, η \in E^{-}} E (t e + η) . \end{matrix}$ By virtue of Lemmas 3.1 and 3.3, there exists $z^{'} \in K$ such that $\begin{matrix} E (z^{'}) ⩽ c \in [δ, sup_{u \in Q (e, R)} E (u)] . \end{matrix}$ Meanwhile, $\begin{matrix} sup_{u \in Q (e, R)} E (u) ⩽ max_{t ⩾ 0, η \in E^{-}} E (t e + η) < c_{1}, \end{matrix}$ which is a contradiction. Thus $c_{1} = {inf}_{z \in N} E (z)$ . □

Choose $K > 0$ such that $\begin{matrix} (3.10) & K > {(\frac{β_{0} c_{1}}{(2 + α) π})}^{p - 1} . \end{matrix}$ Consider the following system $\begin{matrix} (3.11) & \{\begin{array}{l} - Δ u + u = K (I_{α} * | v |^{p}) | v |^{p - 2} v, x \in R^{2}, \\ - Δ v + v = K (I_{β} * | u |^{p}) | u |^{p - 2} u, x \in R^{2} . \end{array} \end{matrix}$ Note that the corresponding energy functional $\begin{matrix} E_{K} (u) = \int_{R^{2}} (\nabla u \nabla v + u v) d x - \frac{K}{2 p} \int_{R^{2}} [(I_{β} * | u |^{p}) | u |^{p} + (I_{α} * | v |^{p}) | v |^{p}] d x \end{matrix}$ is well defined on E. It is easy to show that $z = (u, v)$ is a solution of (3.1) if and only if $z_{K} : = K^{\frac{1}{2 - 2 p}} z = K^{\frac{1}{2 - 2 p}} (u, v)$ is a solution of (3.11). Moreover $\begin{matrix} E_{K} (z_{K}) = (\frac{1}{2} - \frac{1}{2 p}) K \int_{R^{2}} H (z_{K}) d x = (\frac{1}{2} - \frac{1}{2 p}) K^{\frac{1}{1 - p}} \int_{R^{2}} H (z) d x = K^{\frac{1}{1 - p}} E (z), z \in K . \end{matrix}$ Similarly, we define $\begin{matrix} (3.12) & c_{K} = inf_{z \in N_{K}} E_{K} (z), \end{matrix}$ where $\begin{matrix} N_{K} : = {z \in E ∖ {0} : ⟨ E_{K}^{'} (z), z ⟩ = ⟨ E_{K}^{'} (z), η ⟩ = 0, \forall η \in E^{-}} . \end{matrix}$ Therefore $\begin{matrix} (3.13) & c_{K} = K^{\frac{1}{1 - p}} c_{1} . \end{matrix}$

Similar to Lemma 3.4, we have
Lemma 3.6.

$E_{K} (z) = {max}_{t ⩾ 0, η \in E^{-}} E_{K} (t z^{+} + η)$ , for every $z \in N_{K}$ ;

For any $e \in E^{+} ∖ {0}$ , there exist unique constant $t > 0$ and $η \in E^{-}$ such that $t e + η \in N_{K}$ .

Lemma 3.7.
${inf}_{M} Φ (z) ⩽ c_{K}$ .
Proof.
Suppose that $z = z^{+} + z^{-}$ is a solution of (3.11) satisfying $E_{K} (z) = c_{K}$ . Note that $z^{+} \neq 0$ , otherwise, $E_{K} (z) ⩽ 0$ . By using Lemma 2.6, there exist $t > 0$ , $η \in E^{-}$ such that $t z^{+} + η \in M$ . From Lemma 3.6 and $z \in N_{K}$ , we have $\begin{matrix} E_{K} (z) ⩾ E_{K} (t z^{+} + η) . \end{matrix}$ We deduce from (F4) that $\begin{matrix} c_{K} = E_{K} (z) ⩾ E_{K} (t z^{+} + η) ⩾ Φ (t z^{+} + η) ⩾ inf_{M} Φ . \end{matrix}$ Then $\begin{matrix} c_{K} = K^{\frac{1}{1 - p}} c_{1} ⩾ inf_{M} Φ . \end{matrix}$ □

By (3.10) and Lemma 3.7, we have the following lemma. Lemma 3.8.
Assume that (F1), (F2), (F4) and (F5) hold. Then $\begin{matrix} b = inf_{z \in M} Φ (z) < \frac{(2 + α) π}{β_{0}} . \end{matrix}$

4. Proof of main result

In this section, we shall find the ground state solution for the system (1.1) without the classical Ambrosetti–Rabinowitz condition. As mentioned in the introduction, we need to overcome the difficulties caused by the critical exponential growth of nonlinearity, strongly indefinite characteristic and lack of strictly monotonicity condition about f. In order to achieve our goal, we will employ an approximation scheme and the concentration compactness argument, and apply the non-Nehari manifold method.

Define $\begin{matrix} E_{k}^{-} = span {(e_{1}, - e_{1}), (e_{2}, - e_{2}), \dots, (e_{k}, - e_{k})}, E_{k} = E_{k}^{-} \oplus E^{+}, k \in N, \end{matrix}$ where ${e_{k}}$ is a total orthonormal sequence in $H^{1} (R^{2})$ .

Lemma 4.1 ([7]).

Let $X = Y \oplus Z$ be a $Banach$ space with $dim Y < \infty$ , Let $e \in Z$ , with $‖ e ‖ = 1$ and let $0 < ρ < R$ be given positive real number. Set $\begin{matrix} S_{ρ} = {u \in Z : ‖ u ‖ = ρ}, \tilde{Q} = {v + s e : v \in Y, ‖ v ‖ ⩽ R, 0 ⩽ s ⩽ R} . \end{matrix}$ Assume that $Φ \in C^{1} (X, R)$ satisfying $\begin{matrix} inf_{S_{ρ}} Φ > sup_{\partial \tilde{Q}} Φ . \end{matrix}$ Then there exists a sequence ${u_{n}} \subset X$ satisfying $\begin{matrix} Φ (u_{n}) \to c, ‖ Φ^{'} (u_{n}) ‖ (1 + ‖ u_{n} ‖) \to 0, \end{matrix}$ with $\begin{matrix} c = inf_{γ \in Γ} sup_{u \in \tilde{Q}} Φ (γ (u)), \end{matrix}$ where $\begin{matrix} Γ = {γ \in C (\tilde{Q}, X) : γ |_{\partial \tilde{Q}} = id} . \end{matrix}$

For some $r_{0} > 0$ , define $\begin{matrix} Q_{k} = {(w, - w) + s (e, e) : (w, - w) \in E_{k}^{-}, 0 ⩽ s ⩽ r_{0}, ‖ w ‖ ⩽ r_{0}}, k \in N . \end{matrix}$ Note that for the set Q defined by (2.14), there holds $\partial Q_{k} \subset \partial Q$ . Then by Lemma 2.3, we have following lemma.

Lemma 4.2.
Suppose that $(F 1)$ and $(F 2)$ hold with $F (t) ⩾ 0$ for any $t \in R$ . Let $(e, e) \in E^{+}$ with $‖ (e, e) ‖ = 1$ . Then there exists $r_{0} > \bar{ρ}$ such that $\begin{matrix} sup Φ (\partial Q_{k}) ⩽ 0, \end{matrix}$ where the constant $\bar{ρ} > 0$ is given by Lemma 2.2 .

Combining Corollaries 2.5, 2.7 with Lemmas 2.2, 4.1, 4.2, we can show the following lemma similarly as [41, Lemma 2.10].
Lemma 4.3.
Suppose that $(F 1), (F 2)$ and $(F 5)$ hold with $F (t) ⩾ 0$ for all $t \in R$ . Then for every $k \in N$ , there exist ${z_{n}^{k}} \subset E_{k}$ and ${\bar{c}}_{k} \in (0, b_{k}]$ such that $\begin{matrix} (4.1) & Φ (z_{n}^{k}) \to {\bar{c}}_{k}, {‖ Φ^{'} (z_{n}^{k}) ‖}_{E_{k}^{}} (1 + ‖ z_{n}^{k} ‖) \to 0, \end{matrix}$ where* $\begin{matrix} b_{k} : = inf_{M_{k}} Φ = inf_{z \in E_{k} ∖ E_{k}^{-}} max_{t ⩾ 0, ζ \in E_{k}^{-}} Φ (t z + ζ), \end{matrix}$ and $\begin{matrix} M_{k} : = {z \in E_{k} ∖ E_{k}^{-} : ⟨ Φ^{'} (z), z ⟩ = ⟨ Φ^{'} (z), ζ ⟩ = 0, \forall ζ \in E_{k}^{-}} . \end{matrix}$

Note that $E_{k}^{-} \subset E^{-}$ , we deduce the following result from Corollary 2.7 and Lemmas 3.8 and 4.3.
Lemma 4.4.
Let $(F 1), (F 2), (F 4)$ and $(F 5)$ be satisfied. There holds $\begin{matrix} (4.2) & {\bar{c}}_{k} ⩽ b_{k} ⩽ b < \frac{(2 + α) π}{β_{0}} . \end{matrix}$

In order to find nontrivial solutions for (1.1), we first show the boundedness of sequence ${z_{n}^{k}}$ for every $k \in N$ .
Lemma 4.5.
Let $(F 1), (F 2)$ and $(F 5)$ be satisfied. Then the sequence ${z_{n}^{k}}$ given by ( 4.1 ) is bounded in E.
Proof.
Choose $(ϕ, ψ) = (u_{n}^{k}, v_{n}^{k})$ . By (4.1) and $(F 5)$ , we have $\begin{aligned} {\bar{c}}_{k} + o_{n} (1) \\ = Φ (z_{n}^{k}) - \frac{1}{2} ⟨ Φ^{'} (z_{n}^{k}), z_{n}^{k} ⟩ \\ = \frac{1}{2} \int_{R^{2}} [(I_{α} * F (v_{n}^{k})) (f (v_{n}^{k}) v_{n}^{k} - F (v_{n}^{k})) + (I_{β} * F (u_{n}^{k})) (f (u_{n}^{k}) u_{n}^{k} - F (u_{n}^{k}))] d x \\ ⩾ \frac{1}{4} \int_{R^{2}} [(I_{α} * F (v_{n}^{k})) f (v_{n}^{k}) v_{n}^{k} + (I_{β} * F (u_{n}^{k})) f (u_{n}^{k}) u_{n}^{k}] d x \\ (4.3) & ⩾ \frac{1}{2} \int_{R^{2}} [(I_{α} * F (v_{n}^{k})) F (v_{n}^{k}) + (I_{β} * F (u_{n}^{k})) F (u_{n}^{k})] d x . \end{aligned}$ Using (2.5) and (4.1), and choosing $(ϕ, ψ) = (v_{n}^{k}, 0)$ and $(ϕ, ψ) = (0, u_{n}^{k})$ , one has $\begin{array}{r} (4.4) & {‖ v_{n}^{k} ‖}^{2} = \int_{R^{2}} (I_{β} * F (u_{n}^{k})) f (u_{n}^{k}) v_{n}^{k} d x + o_{n} (1), \end{array}$ and $\begin{array}{r} (4.5) & {‖ u_{n}^{k} ‖}^{2} = \int_{R^{2}} (I_{α} * F (v_{n}^{k})) f (v_{n}^{k}) u_{n}^{k} d x + o_{n} (1) . \end{array}$ We assume, without loss of generality, that $‖ u_{n}^{k} ‖, ‖ v_{n}^{k} ‖ ⩾ 1$ for n large. Let $U_{n}^{k} = u_{n}^{k} / ‖ u_{n}^{k} ‖$ and $V_{n}^{k} = v_{n}^{k} / ‖ v_{n}^{k} ‖$ , there hold $\begin{array}{r} (4.6) & ‖ v_{n}^{k} ‖ = \int_{R^{2}} (I_{β} * F (u_{n}^{k})) f (u_{n}^{k}) V_{n}^{k} d x + o_{n} (1), \end{array}$ and $\begin{array}{r} (4.7) & ‖ u_{n}^{k} ‖ = \int_{R^{2}} (I_{α} * F (v_{n}^{k})) f (v_{n}^{k}) U_{n}^{k} d x + o_{n} (1) . \end{array}$ In view of $(F 1)$ and $(F 2)$ , there exist $\tilde{β} > β_{0} > 0$ and $C_{1} > 0$ such that $\begin{array}{r} (4.8) & | f (t) | ⩽ C_{1} e^{\tilde{β} t^{2}}, \forall t ⩾ 0 . \end{array}$ Moreover, there exists $C_{2} > 0$ such that $\begin{array}{r} (4.9) & f^{2} (t) ⩽ C_{2} f (t) t, when \frac{f (t)}{C_{1}} ⩽ e^{\frac{1}{4}} . \end{array}$

Set $\begin{matrix} Ω_{n}^{1} = {x \in R^{2} : \frac{f (u_{n}^{k})}{C_{1}} ⩽ e^{\frac{1}{4}}}, Ω_{n}^{2} = {x \in R^{2} : \frac{f (u_{n}^{k})}{C_{1}} ⩾ e^{\frac{1}{4}}} . \end{matrix}$ Applying Lemma 2.1 with $t = V_{n}^{k}$ , $s = f (u_{n}^{k}) / C_{1}$ , one has $\begin{array}{rcl} ‖ v_{n}^{k} ‖ & = & C_{1} \int_{R^{2}} (I_{β} * F (u_{n}^{k})) \frac{f (u_{n}^{k})}{C_{1}} V_{n}^{k} d x + o_{n} (1) \\ ⩽ & C_{1} \int_{R^{2}} (I_{β} * F (u_{n}^{k})) (e^{{(V_{n}^{k})}^{2}} - 1) d x \\ + \frac{1}{2 C_{1}} \int_{Ω_{n}^{1}} (I_{β} * F (u_{n}^{k})) {(f (u_{n}^{k}))}^{2} d x \\ + C_{1} \int_{Ω_{n}^{2}} (I_{β} * F (u_{n}^{k})) \frac{f (u_{n}^{k})}{C_{1}} {(log \frac{f (u_{n}^{k})}{C_{1}})}^{\frac{1}{2}} d x + o_{n} (1) \\ (4.10) & : = & I_{1} + I_{2} + I_{3} + o_{n} (1) . \end{array}$ From (1.6), (1.8) and Lemma 1.1-ii), we deduce that $\begin{array}{rcl} I_{1} & = & C_{1} \int_{R^{2}} (I_{β} * F (u_{n}^{k})) (e^{{(V_{n}^{k})}^{2}} - 1) d x \\ ⩽ & C_{1} {[\int_{R^{2}} (I_{β} * F (u_{n}^{k})) F (u_{n}^{k}) d x]}^{\frac{1}{2}} \\ \times {[\int_{R^{2}} (I_{β} * (e^{{(V_{n}^{k})}^{2}} - 1)) (e^{{(V_{n}^{k})}^{2}} - 1) d x]}^{\frac{1}{2}} \\ ⩽ & C_{1} {(2 {\bar{c}}_{k} + 1)}^{\frac{1}{2}} {‖ e^{{(V_{n}^{k})}^{2}} - 1 ‖}_{4 / (2 + β)} \\ (4.11) & ⩽ & C_{1} C_{3} {(2 {\bar{c}}_{k} + 1)}^{\frac{1}{2}} . \end{array}$ In view of (4.3), (4.8) and (4.29), we have $\begin{array}{rcl} I_{2} & = & \frac{1}{2 C_{1}} \int_{Ω_{n}^{1}} (I_{β} * F (u_{n}^{k})) {(f (u_{n}^{k}))}^{2} d x \\ ⩽ & \frac{C_{2}}{2 C_{1}} \int_{R^{2}} (I_{β} * F (u_{n}^{k})) f (u_{n}^{k}) u_{n}^{k} d x \\ (4.12) & ⩽ & \frac{C_{2}}{2 C_{1}} (4 {\bar{c}}_{k} + 1), \end{array}$ and $\begin{array}{rcl} I_{3} & = & C_{1} \int_{Ω_{n}^{2}} (I_{β} * F (u_{n}^{k})) \frac{f (u_{n}^{k})}{C_{1}} {(log \frac{f (u_{n}^{k})}{C_{1}})}^{\frac{1}{2}} d x \\ ⩽ & C_{4} \int_{Ω_{n}^{2}} (I_{β} * F (u_{n}^{k})) f (u_{n}^{k}) u_{n}^{k} d x \\ (4.13) & ⩽ & C_{4} (4 {\bar{c}}_{k} + 1) . \end{array}$ Then it follows from (4.2), (4.11), (4.12) and (4.13) that $\begin{array}{r} (4.14) & ‖ v_{n}^{k} ‖ ⩽ C_{6}, \end{array}$ Similarly, we show that $\begin{array}{r} (4.15) & ‖ u_{n}^{k} ‖ ⩽ C_{7} . \end{array}$ Thus, $z_{n}^{k} = (u_{n}^{k}, v_{n}^{k})$ is bounded in E. □

The following lemma shows that the Fréchet derivative of functional $\int_{R^{2}} (I_{α} * | F (u) |) F (u) d x$ has weak-to- $wea k^{*}$ sequentially continuous with the aid of the following auxiliary condition (4.16).
Lemma 4.6 ([37, Lemma 4.8]).

Let $(F 1), (F 2), (F 3)$ and $(F 5)$ be satisfied. Assume that $(u_{n}, v_{n}) ⇀ (\bar{u}, \bar{v})$ in E and for some constant $K_{0} > 0$ , $\begin{array}{r} (4.16) & \int_{R^{2}} (I_{α} * F (v_{n})) f (v_{n}) v_{n} d x ⩽ K_{0}, \int_{R^{2}} (I_{β} * F (u_{n})) f (u_{n}) u_{n} d x ⩽ K_{0} . \end{array}$ Then for every $ϕ, ψ \in C_{0}^{\infty} (R^{2})$ , we have $\begin{array}{r} lim_{n \to \infty} \int_{R^{2}} (I_{α} * F (v_{n})) f (v_{n}) ψ d x = \int_{R^{2}} (I_{α} * F (v)) f (v) ψ d x, \end{array}$ and $\begin{array}{r} lim_{n \to \infty} \int_{R^{2}} (I_{β} * F (u_{n})) f (u_{n}) ϕ d x = \int_{R^{2}} (I_{β} * F (u)) f (u) ϕ d x . \end{array}$

Proof of Theorem 1.4.

By Lemmas 4.3 and 4.4, there exist subsequences ${{\bar{c}}_{k_{n}}} \subset {{\bar{c}}_{k}}$ and ${z_{j_{n}}^{k_{n}}} \subset E_{k_{n}}$ such that $\begin{matrix} lim_{n \to \infty} {\bar{c}}_{k_{n}} = \bar{c} \in [κ_{0}, b] \subset [κ_{0}, (2 + α) π / β_{0}), \end{matrix}$ and $\begin{matrix} Φ (z_{j_{n}}^{k_{n}}) \to \bar{c}, {‖ Φ^{'} (z_{j_{n}}^{k_{n}}) ‖}_{E_{k_{n}}^{*}} (1 + ‖ {\tilde{z}}_{j_{n}}^{k_{n}} ‖) \to 0 . \end{matrix}$ For notational simplicity, set ${\tilde{z}}_{n} = z_{j_{n}}^{k_{n}}$ and ${\tilde{z}}_{n} = ({\tilde{u}}_{n}, {\tilde{v}}_{n})$ , then we have $\begin{array}{r} (4.17) & Φ ({\tilde{z}}_{n}) \to \bar{c} \in [κ_{0}, b], {‖ Φ^{'} ({\tilde{z}}_{n}) ‖}_{E_{k_{n}}^{*}} (1 + ‖ {\tilde{z}}_{n} ‖) \to 0 . \end{array}$ Lemma 4.5 yields the existence of a constant $C_{1} > 0$ such that $‖ {\tilde{z}}_{n} ‖ + ‖ {\tilde{u}}_{n} ‖_{2} + ‖ {\tilde{v}}_{n} ‖_{2} ⩽ C_{1}$ . By virtue of $(F 5)$ , (2.7) and (4.17), we have $\begin{array}{r} (4.18) & 2 \int_{R^{2}} (I_{α} * F ({\tilde{v}}_{n})) F ({\tilde{v}}_{n}) d x ⩽ \int_{R^{2}} (I_{α} * F ({\tilde{v}}_{n})) f ({\tilde{v}}_{n}) {\tilde{v}}_{n} d x ⩽ C_{2}, \end{array}$ and $\begin{array}{r} (4.19) & 2 \int_{R^{2}} (I_{β} * F ({\tilde{u}}_{n})) F ({\tilde{u}}_{n}) d x ⩽ \int_{R^{2}} (I_{β} * F ({\tilde{u}}_{n})) f ({\tilde{u}}_{n}) {\tilde{u}}_{n} d x ⩽ C_{2} . \end{array}$ If $\begin{matrix} δ_{0} : = \underset{n \to \infty}{lim sup} sup_{y \in R^{2}} \int_{B_{1} (y)} (| {\tilde{u}}_{n} |^{2} + | {\tilde{v}}_{n} |^{2}) d x = 0, \end{matrix}$ then the Lions’ concentration compactness principle (cf. [43, Lemma 1.21]) implies that ${\tilde{u}}_{n} \to 0$ , ${\tilde{v}}_{n} \to 0$ in $L^{s} (R^{2})$ for $s \in (2, \infty)$ . For any given $ε > 0$ , set $M = M (ε) > M_{0} C_{2} / ε$ . Then by $(F 3)$ and (4.19), we have $\begin{array}{rcl} \int_{| {\tilde{u}}_{n} | ⩾ M} (I_{β} * F ({\tilde{u}}_{n})) F ({\tilde{u}}_{n}) d x & ⩽ & M_{0} \int_{| {\tilde{u}}_{n} | ⩾ M} (I_{β} * F ({\tilde{u}}_{n})) f ({\tilde{u}}_{n}) d x \\ ⩽ & \frac{M_{0}}{M} \int_{| {\tilde{u}}_{n} | ⩾ M} (I_{β} * F ({\tilde{u}}_{n})) f ({\tilde{u}}_{n}) {\tilde{u}}_{n} d x \\ (4.20) & < & ε . \end{array}$ Choosing $N = N (ε) \in (0, 1)$ , it follows from $(F 2)$ , (1.6), (1.8), (2.2) and (4.19) that $\begin{aligned} \int_{| {\tilde{u}}_{n} | ⩽ N_{ε}} (I_{β} * F ({\tilde{u}}_{n})) F ({\tilde{u}}_{n}) d x \\ ⩽ \int_{| {\tilde{u}}_{n} | ⩽ N_{ε}} (I_{β} * F ({\tilde{u}}_{n})) f ({\tilde{u}}_{n}) {\tilde{u}}_{n} d x \\ ⩽ \frac{ε}{C_{1}^{2} γ_{8 / (2 + β)}^{2}} \int_{| {\tilde{u}}_{n} | ⩽ N_{ε}} (I_{β} * F ({\tilde{u}}_{n})) | {\tilde{u}}_{n} |^{2} d x \\ ⩽ \frac{ε}{C_{1}^{2} γ_{8 / (2 + β)}^{2}} {(\int_{R^{2}} (I_{β} * F ({\tilde{u}}_{n})) F ({\tilde{u}}_{n}) d x)}^{\frac{1}{2}} {(\int_{R^{2}} (I_{β} * | {\tilde{u}}_{n} |^{2}) | {\tilde{u}}_{n} |^{2} d x)}^{\frac{1}{2}} \\ ⩽ \frac{\sqrt{C_{2} C_{0}} ε}{C_{1}^{2} γ_{8 / (2 + β)}^{2}} ‖ {\tilde{u}}_{n} ‖_{8 / (2 + β)}^{2} \\ (4.21) & = o (1) . \end{aligned}$ By $(F 1)$ , $(F 2)$ , (1.6), (1.8) and (4.19), we get $\begin{aligned} \int_{N ⩽ | {\tilde{u}}_{n} | ⩽ M} (I_{β} * F ({\tilde{u}}_{n})) F ({\tilde{u}}_{n}) d x \\ ⩽ \int_{N ⩽ | {\tilde{u}}_{n} | ⩽ M} (I_{β} * F ({\tilde{u}}_{n})) f ({\tilde{u}}_{n}) {\tilde{u}}_{n} d x \\ ⩽ C_{3} \int_{N ⩽ | {\tilde{u}}_{n} | ⩽ M} (I_{β} * F ({\tilde{u}}_{n})) | {\tilde{u}}_{n} |^{3} d x \\ ⩽ C_{3} {(\int_{N ⩽ | {\tilde{u}}_{n} | ⩽ M} (I_{β} * F ({\tilde{u}}_{n})) F ({\tilde{u}}_{n}) d x)}^{\frac{1}{2}} {(\int_{N ⩽ | {\tilde{u}}_{n} | ⩽ M} (I_{β} * | {\tilde{u}}_{n} |^{3}) | {\tilde{u}}_{n} |^{3} d x)}^{\frac{1}{2}} \\ ⩽ C_{4} ‖ {\tilde{u}}_{n} ‖_{12 / (2 + β)}^{3} \\ (4.22) & = o (1) . \end{aligned}$ Since $ε > 0$ is arbitrary, it follows from (4.20), (4.21) and (4.22) that $\begin{array}{r} (4.23) & \int_{R^{2}} (I_{β} * F ({\tilde{u}}_{n})) F ({\tilde{u}}_{n}) d x = o (1) . \end{array}$ Similarly, we get $\begin{array}{r} (4.24) & \int_{R^{2}} (I_{α} * F ({\tilde{v}}_{n})) F ({\tilde{v}}_{n}) d x = o (1) . \end{array}$ From (2.4), (2.5) and (4.17), we deduce that $\begin{aligned} \int_{R^{2}} (\nabla {\tilde{u}}_{n} \nabla {\tilde{v}}_{n} + V (x) {\tilde{u}}_{n} {\tilde{v}}_{n}) d x \\ = \bar{c} + \frac{1}{2} \int_{R^{2}} (I_{α} * F ({\tilde{v}}_{n})) F ({\tilde{v}}_{n}) d x + \frac{1}{2} \int_{R^{2}} (I_{β} * F ({\tilde{u}}_{n})) F ({\tilde{u}}_{n}) d x + o (1) \\ (4.25) & = \bar{c} + o (1) : = \frac{(2 + α) π}{β_{0}} (1 - 4 \bar{ε}) + o (1), \end{aligned}$ and $\begin{aligned} 2 \int_{R^{2}} (\nabla {\tilde{u}}_{n} \nabla {\tilde{v}}_{n} + V (x) {\tilde{u}}_{n} {\tilde{v}}_{n}) d x \\ = ⟨ Φ^{'} ({\tilde{z}}_{n}), {\tilde{z}}_{n} ⟩ + \int_{R^{2}} (I_{α} * F ({\tilde{v}}_{n})) f ({\tilde{v}}_{n}) {\tilde{v}}_{n} d x + \int_{R^{2}} (I_{β} * F ({\tilde{u}}_{n})) f ({\tilde{u}}_{n}) {\tilde{u}}_{n} d x \\ (4.26) & = \int_{R^{2}} (I_{α} * F ({\tilde{v}}_{n})) f ({\tilde{v}}_{n}) {\tilde{v}}_{n} d x + \int_{R^{2}} (I_{β} * F ({\tilde{u}}_{n})) f ({\tilde{u}}_{n}) {\tilde{u}}_{n} d x + o (1), \end{aligned}$ here $\bar{ε} > 0$ is small enough. Choosing the test functions $(ϕ, ψ) = ({\tilde{v}}_{n}, 0)$ , $(ϕ, ψ) = (0, {\tilde{u}}_{n})$ in (2.5) respectively, by (4.17) we have $\begin{array}{r} (4.27) & ‖ {\tilde{v}}_{n} ‖^{2} = \int_{R^{2}} (I_{β} * F ({\tilde{u}}_{n})) f ({\tilde{u}}_{n}) {\tilde{v}}_{n} d x + o (1), \end{array}$ and $\begin{array}{r} (4.28) & ‖ {\tilde{u}}_{n} ‖^{2} = \int_{R^{2}} (I_{α} * F ({\tilde{v}}_{n})) f ({\tilde{v}}_{n}) {\tilde{u}}_{n} d x + o (1) . \end{array}$ Set $\begin{matrix} {\tilde{U}}_{n} = {[(2 + α) π (1 - 4 \bar{ε})]}^{\frac{1}{2}} \frac{{\tilde{u}}_{n}}{‖ {\tilde{u}}_{n} ‖} . \end{matrix}$ It follows from (F1) that there exist constants $C_{\bar{ε}}, C_{5} > 0$ such that $\begin{array}{r} f (t) ⩽ C_{\bar{ε}} e^{β_{0} (1 + \bar{ε}) t^{2}}, \forall t ⩾ 0, \end{array}$ and $\begin{array}{r} f (t) ⩽ C_{5} t when \frac{f ({\tilde{v}}_{n})}{C_{\bar{ε}}} ⩽ e^{\frac{1}{4}} . \end{array}$ Applying Lemma 2.1 with $s = f ({\tilde{v}}_{n}) / C_{\bar{ε}}$ and $t = {\tilde{U}}_{n}$ , and combining Lemma 1.1-ii) with (1.6), (1.8) and (4.24), we have $\begin{aligned} {[(2 + α) π (1 - 4 \bar{ε})]}^{\frac{1}{2}} ‖ {\tilde{u}}_{n} ‖ \\ = \int_{R^{2}} (I_{α} * F ({\tilde{v}}_{n})) f ({\tilde{v}}_{n}) {\tilde{U}}_{n} d x + o (1) \\ = C_{\bar{ε}} \int_{R^{2}} (I_{α} * F ({\tilde{v}}_{n})) \frac{f ({\tilde{v}}_{n})}{C_{\bar{ε}}} {\tilde{U}}_{n} d x + o (1) \\ ⩽ C_{\bar{ε}} \int_{R^{2}} (I_{α} * F ({\tilde{v}}_{n})) (e^{{\tilde{U}}_{n}^{2}} - 1) d x \\ + \frac{C_{\bar{ε}}}{2} \int_{\frac{f ({\tilde{v}}_{n})}{C_{\bar{ε}}} ⩽ e^{\frac{1}{4}}} (I_{α} * F ({\tilde{v}}_{n})) {(\frac{f ({\tilde{v}}_{n})}{C_{\bar{ε}}})}^{2} d x \\ + C_{\bar{ε}} \int_{\frac{f ({\tilde{v}}_{n})}{C_{\bar{ε}}} ⩾ e^{\frac{1}{4}}} (I_{α} * F ({\tilde{v}}_{n})) \frac{f ({\tilde{v}}_{n})}{C_{\bar{ε}}} {(log \frac{f ({\tilde{v}}_{n})}{C_{\bar{ε}}})}^{\frac{1}{2}} d x + o (1) \\ ⩽ C_{\bar{ε}} {[\int_{R^{2}} (I_{α} * F ({\tilde{v}}_{n})) F ({\tilde{v}}_{n}) d x]}^{\frac{1}{2}} {[\int_{R^{2}} (I_{α} * (e^{{\tilde{U}}_{n}^{2}} - 1)) (e^{{\tilde{U}}_{n}^{2}} - 1) d x]}^{\frac{1}{2}} \\ + \frac{C_{0}}{2 C_{\bar{ε}}} {[\int_{\frac{f ({\tilde{v}}_{n})}{C_{\bar{ε}}} ⩽ e^{\frac{1}{4}}} (I_{α} * F ({\tilde{v}}_{n})) F ({\tilde{v}}_{n}) d x]}^{\frac{1}{2}} {[\int_{\frac{f ({\tilde{v}}_{n})}{C_{\bar{ε}}} ⩽ e^{\frac{1}{4}}} (I_{α} * f^{2} ({\tilde{v}}_{n})) f^{2} ({\tilde{v}}_{n}) d x]}^{\frac{1}{2}} \\ + {[β_{0} (1 + \bar{ε})]}^{\frac{1}{2}} \int_{\frac{f ({\tilde{v}}_{n})}{C_{\bar{ε}}} ⩾ e^{\frac{1}{4}}} (I_{α} * F ({\tilde{v}}_{n})) f ({\tilde{v}}_{n}) {\tilde{v}}_{n} d x + o (1) \\ ⩽ C_{0} C_{\bar{ε}} {[\int_{R^{2}} (I_{α} * F ({\tilde{v}}_{n})) F ({\tilde{v}}_{n}) d x]}^{\frac{1}{2}} {[\int_{R^{2}} {(e^{{\tilde{U}}_{n}^{2}} - 1)}^{\frac{4}{2 + α}} d x]}^{\frac{2 + α}{4}} \\ + \frac{C_{0} C_{5}^{4}}{2 C_{\bar{ε}}} {[\int_{\frac{f ({\tilde{v}}_{n})}{C_{\bar{ε}}} ⩽ e^{\frac{1}{4}}} (I_{α} * F ({\tilde{v}}_{n})) F ({\tilde{v}}_{n}) d x]}^{\frac{1}{2}} {[\int_{\frac{f ({\tilde{v}}_{n})}{C_{\bar{ε}}} ⩽ e^{\frac{1}{4}}} | {\tilde{v}}_{n} |^{8 / (2 + α)} d x]}^{\frac{2 + α}{4}} \\ + {[β_{0} (1 + \bar{ε})]}^{\frac{1}{2}} \int_{R^{2}} (I_{α} * F ({\tilde{v}}_{n})) f ({\tilde{v}}_{n}) {\tilde{v}}_{n} d x + o (1) \\ ⩽ C_{0} C_{\bar{ε}} {[\int_{R^{2}} (I_{α} * F ({\tilde{v}}_{n})) F ({\tilde{v}}_{n}) d x]}^{\frac{1}{2}} {[\int_{R^{2}} (e^{4 π (1 - 4 \bar{ε}) \frac{{\tilde{u}}_{n}^{2}}{‖ {\tilde{u}}_{n} ‖^{2}}} - 1) d x]}^{\frac{2 + α}{4}} \\ + \frac{C_{0} C_{5}^{4}}{2 C_{\bar{ε}}} {[\int_{R^{2}} (I_{α} * F ({\tilde{v}}_{n})) F ({\tilde{v}}_{n}) d x]}^{\frac{1}{2}} ‖ {\tilde{v}}_{n} ‖_{8 / (2 + α)}^{2} \\ + {[β_{0} (1 + \bar{ε})]}^{\frac{1}{2}} \int_{R^{2}} (I_{α} * F ({\tilde{v}}_{n})) f ({\tilde{v}}_{n}) {\tilde{v}}_{n} d x + o (1) \\ (4.29) & = {[β_{0} (1 + \bar{ε})]}^{\frac{1}{2}} \int_{R^{2}} (I_{α} * F ({\tilde{v}}_{n})) f ({\tilde{v}}_{n}) {\tilde{v}}_{n} d x + o (1) . \end{aligned}$

Similarly, we get $\begin{array}{r} (4.30) & {[(2 + α) π (1 - 4 \bar{ε})]}^{\frac{1}{2}} ‖ {\tilde{v}}_{n} ‖ ⩽ {[β_{0} (1 + \bar{ε})]}^{\frac{1}{2}} \int_{R^{2}} [I_{β} * F ({\tilde{u}}_{n})] f ({\tilde{u}}_{n}) {\tilde{u}}_{n} d x + o (1) . \end{array}$ Then we deduce from (4.25), (4.26) (4.29) and (4.30) that $\begin{aligned} {[(2 + α) π (1 - 4 \bar{ε})]}^{\frac{1}{2}} (‖ {\tilde{u}}_{n} ‖ + ‖ {\tilde{v}}_{n} ‖) \\ ⩽ {[β_{0} (1 + \bar{ε})]}^{\frac{1}{2}} [\int_{R^{2}} (I_{α} * F ({\tilde{v}}_{n})) f ({\tilde{v}}_{n}) {\tilde{v}}_{n} d x + \int_{R^{2}} (I_{β} * F ({\tilde{u}}_{n})) f ({\tilde{u}}_{n}) {\tilde{u}}_{n} d x] + o (1) \\ ⩽ 2 {[β_{0} (1 + \bar{ε})]}^{\frac{1}{2}} \frac{(2 + α) π}{β_{0}} (1 - 4 \bar{ε}) + o (1) . \end{aligned}$ Therefore $\begin{array}{rcl} ‖ {\tilde{u}}_{n} ‖ + ‖ {\tilde{v}}_{n} ‖ & ⩽ & 2 {[β_{0}^{- 1} (1 + \bar{ε})]}^{\frac{1}{2}} {[(2 + α) π (1 - 4 \bar{ε})]}^{\frac{1}{2}} + o (1) . \end{array}$ Note that there must be a subsequence of ${{\tilde{u}}_{n}}$ or ${{\tilde{v}}_{n}}$ , without loss of generality, we assume it is ${{\tilde{u}}_{n}}$ (still denoted by ${{\tilde{u}}_{n}}$ ) such that $\begin{array}{rcl} (4.31) & ‖ {\tilde{u}}_{n} ‖ & ⩽ & {[β_{0}^{- 1} (1 + \bar{ε})]}^{\frac{1}{2}} {[(2 + α) π (1 - 4 \bar{ε})]}^{\frac{1}{2}} + o (1) . \end{array}$ By $(F 1)$ , there exists $C_{6} > 0$ satisfy $\begin{array}{r} (4.32) & | f (t) | ⩽ C_{6} (e^{β_{0} (1 + \bar{ε}) t^{2}} - 1), \forall | t | ⩾ 1 . \end{array}$ Now, we choose $q \in (1, 2)$ such that $\begin{array}{r} {(1 + \bar{ε})}^{2} (1 - 4 \bar{ε}) q < 1 . \end{array}$ It follows from (4.31), (4.32) and Lemma 1.1-ii) that $\begin{array}{rcl} \int_{| {\tilde{u}}_{n} | ⩾ 1} {| f ({\tilde{u}}_{n}) |}^{\frac{4 q}{2 + β}} d x & ⩽ & C_{7} \int_{| {\tilde{u}}_{n} | ⩾ 1} {(e^{β_{0} (1 + \bar{ε}) {\tilde{u}}_{n}^{2}} - 1)}^{\frac{4 q}{2 + β}} d x \\ ⩽ & C_{7} \int_{R^{2}} [\exp (\frac{4 β_{0} (1 + \bar{ε}) q ‖ {\tilde{u}}_{n} ‖^{2}}{2 + β} \frac{{\tilde{u}}_{n}^{2}}{‖ {\tilde{u}}_{n} ‖^{2}}) - 1] d x \\ ⩽ & C_{7} \int_{R^{2}} [\exp (\frac{(2 + α) {(1 + \bar{ε})}^{2} (1 - 4 \bar{ε}) q 4 π}{2 + β} \frac{{\tilde{u}}_{n}^{2}}{‖ {\tilde{u}}_{n} ‖^{2}}) - 1] d x \\ (4.33) & ⩽ & C_{8} . \end{array}$ Set $q^{'} = q / (q - 1)$ . Then $q^{'} \in (2, + \infty)$ . Using (1.6), (1.8), (4.19), (4.33) and Hölder inequality, we are led to $\begin{aligned} \int_{| {\tilde{u}}_{n} | ⩾ 1} (I_{β} * F ({\tilde{u}}_{n})) f ({\tilde{u}}_{n}) {\tilde{v}}_{n} d x \\ ⩽ {[\int_{R^{2}} (I_{β} * F ({\tilde{u}}_{n})) F ({\tilde{u}}_{n}) d x]}^{\frac{1}{2}} {[\int_{R^{2}} (I_{β} * f ({\tilde{u}}_{n}) {\tilde{v}}_{n} χ_{| \tilde{u_{n}} ⩾ 1 |}) f ({\tilde{u}}_{n}) {\tilde{v}}_{n} χ_{| \tilde{u_{n}} ⩾ 1 |} d x]}^{\frac{1}{2}} \\ ⩽ \sqrt{C_{2} C_{0}} {[\int_{| {\tilde{u}}_{n} | ⩾ 1} {| f ({\tilde{u}}_{n}) |}^{\frac{4}{2 + β}} | {\tilde{v}}_{n} |^{\frac{4}{2 + β}} d x]}^{\frac{2 + β}{4}} \\ ⩽ \sqrt{C_{2} C_{0}} {[\int_{| {\tilde{u}}_{n} | ⩾ 1} {| f ({\tilde{u}}_{n}) |}^{\frac{4 q}{2 + β}} d x]}^{\frac{2 + β}{4 q}} {[\int_{R^{2}} | {\tilde{v}}_{n} |^{\frac{4 q'}{2 + β}} d x]}^{\frac{2 + β}{4 q'}} \\ (4.34) & ⩽ \sqrt{C_{2} C_{0}} C_{8}^{\frac{2 + β}{4 q}} ‖ {\tilde{v}}_{n} ‖_{\frac{4 q'}{2 + β}} = o (1) . \end{aligned}$ By (F1), (F2), (1.6) and (1.8), there holds $\begin{aligned} \int_{| {\tilde{u}}_{n} | < 1} (I_{β} * F ({\tilde{u}}_{n})) f ({\tilde{u}}_{n}) {\tilde{v}}_{n} d x \\ ⩽ C_{9} \int_{| {\tilde{u}}_{n} | < 1} (I_{β} * F ({\tilde{u}}_{n})) | {\tilde{u}}_{n} | {\tilde{v}}_{n} d x \\ ⩽ C_{9} {[\int_{R^{2}} (I_{β} * F ({\tilde{u}}_{n})) F ({\tilde{u}}_{n}) d x]}^{\frac{1}{2}} {[\int_{R^{2}} (I_{β} * | {\tilde{u}}_{n} {\tilde{v}}_{n} |) | {\tilde{u}}_{n} {\tilde{v}}_{n} | d x]}^{\frac{1}{2}} \\ ⩽ \sqrt{C_{2} C_{0}} C_{9} {(\int_{R^{2}} | {\tilde{u}}_{n} |^{\frac{4}{2 + β}} {\tilde{v}}_{n}^{\frac{4}{2 + β}} d x)}^{\frac{2 + β}{4}} \\ (4.35) & ⩽ \sqrt{C_{2} C_{0}} C_{9} ‖ {\tilde{u}}_{n} ‖_{4 / β} ‖ {\tilde{v}}_{n} ‖_{2} = o (1) . \end{aligned}$ Combining (4.27), (4.34) with (4.35), one has $\begin{array}{rcl} ‖ {\tilde{v}}_{n} ‖^{2} & = & \int_{R^{2}} (I_{β} * F ({\tilde{u}}_{n})) f ({\tilde{u}}_{n}) {\tilde{v}}_{n} d x + o (1) \\ = & \int_{| {\tilde{u}}_{n} | < 1} (I_{β} * F ({\tilde{u}}_{n})) f ({\tilde{u}}_{n}) {\tilde{v}}_{n} d x + \int_{| {\tilde{u}}_{n} | ⩾ 1} (I_{β} * F ({\tilde{u}}_{n})) f ({\tilde{u}}_{n}) {\tilde{v}}_{n} d x + o (1) \\ = & o (1), \end{array}$ which together with (4.25) yields that $\begin{array}{rcl} \bar{c} & = & lim_{n \to \infty} \int_{R^{2}} [\nabla {\tilde{u}}_{n} \nabla {\tilde{v}}_{n} + V (x) {\tilde{u}}_{n} {\tilde{v}}_{n}] d x \\ ⩽ & lim_{n \to \infty} ‖ {\tilde{u}}_{n} ‖ ‖ {\tilde{v}}_{n} ‖ = 0 . \end{array}$ This contradicts with $\bar{c} ⩾ κ_{0} > 0$ . Thus $δ > 0$ . Going if necessary to a subsequence, we assume that there exists ${y_{n}} \subset Z^{2}$ satisfying $\begin{array}{r} (4.36) & \int_{B_{1 + \sqrt{2}} (y_{n})} (| {\tilde{u}}_{n} |^{2} + | {\tilde{v}}_{n} |^{2}) d x > \frac{δ}{2} . \end{array}$ Let ${\hat{u}}_{n} (x) = {\tilde{u}}_{n} (x + y_{n})$ and ${\hat{v}}_{n} (x) = {\tilde{v}}_{n} (x + y_{n})$ , then $\begin{array}{r} (4.37) & \int_{B_{1 + \sqrt{2}} (0)} (| {\hat{u}}_{n} |^{2} + | {\hat{v}}_{n} |^{2}) d x > \frac{δ}{2} . \end{array}$ Set ${\hat{z}}_{n} = ({\hat{u}}_{n}, {\hat{v}}_{n})$ , we have $‖ {\hat{u}}_{n} ‖ = ‖ {\tilde{u}}_{n} ‖$ , $‖ {\hat{v}}_{n} ‖ = ‖ {\tilde{v}}_{n} ‖$ and $\begin{array}{r} (4.38) & Φ ({\hat{z}}_{n}) \to \bar{c}, {‖ Φ^{'} ({\hat{z}}_{n}) ‖}_{E_{k_{n}}^{*}} (1 + ‖ {\hat{z}}_{n} ‖) \to 0 . \end{array}$ Passing to a subsequence, we assume that $({\hat{u}}_{n}, {\hat{v}}_{n}) ⇀ (\hat{u}, \hat{v})$ in E, ${\hat{u}}_{n} \to \hat{u}$ , ${\hat{v}}_{n} \to \hat{v}$ in $L_{loc}^{s} (R^{2}), s \in [2, \infty)$ and ${\hat{u}}_{n} \to \hat{u}$ , ${\hat{v}}_{n} \to \hat{v}$ a.e. $R^{2}$ . By (4.37), we have $(\hat{u}, \hat{v}) \neq (0, 0)$ .

For any $ϕ, ψ \in C_{0}^{\infty} (R^{2})$ , we have $\begin{array}{r} (4.39) & ϕ = \sum_{j = 1}^{\infty} (ϕ, e_{j}) e_{j}, ‖ ϕ ‖^{2} = \sum_{j = 1}^{\infty} {| (ϕ, e_{j}) |}^{2}, \end{array}$ and $\begin{array}{r} (4.40) & ψ = \sum_{j = 1}^{\infty} (ψ, e_{j}) e_{j}, ‖ ψ ‖^{2} = \sum_{j = 1}^{\infty} {| (ψ, e_{j}) |}^{2} . \end{array}$ Set $\begin{array}{r} (4.41) & ϕ_{n} = \sum_{j = 1}^{k_{n}} (ϕ, e_{j}) e_{j}, {\tilde{ϕ}}_{n} = \sum_{k_{n} + 1}^{\infty} (ϕ, e_{j}) e_{j}, \end{array}$ and $\begin{array}{r} (4.42) & ψ_{n} = \sum_{j = 1}^{k_{n}} (ψ, e_{j}) e_{j}, {\tilde{ψ}}_{n} = \sum_{k_{n} + 1}^{\infty} (ψ, e_{j}) e_{j} . \end{array}$ For any given $ε > 0$ , we have $\begin{aligned} \int_{| {\hat{u}}_{n} | ⩾ C_{2} ‖ ϕ ‖_{\infty} ε^{- 1}} (I_{β} * F ({\hat{u}}_{n})) f ({\hat{u}}_{n}) {\tilde{ϕ}}_{n} d x \\ ⩽ \frac{ε}{C_{2}} \int_{| {\hat{u}}_{n} | ⩾ C_{2} ‖ ϕ ‖_{\infty} ε^{- 1}} (I_{β} * F ({\hat{u}}_{n})) f ({\hat{u}}_{n}) {\hat{u}}_{n} d x \\ (4.43) & < ε . \end{aligned}$ By virtue of (1.6), (1.8), (4.19) and Hölder inequality, we have $\begin{aligned} \int_{| {\hat{u}}_{n} | < C_{2} ‖ ϕ ‖_{\infty} ε^{- 1}} (I_{β} * F ({\hat{u}}_{n})) f ({\hat{u}}_{n}) {\tilde{ϕ}}_{n} d x \\ ⩽ {[\int_{R^{2}} (I_{β} * F ({\hat{u}}_{n})) F ({\hat{u}}_{n}) d x]}^{\frac{1}{2}} {[\int_{| {\hat{u}}_{n} | < C_{2} ‖ ϕ ‖_{\infty} ε^{- 1}} (I_{β} * f ({\hat{u}}_{n}) {\tilde{ϕ}}_{n}) f ({\hat{u}}_{n}) {\tilde{ϕ}}_{n} d x]}^{\frac{1}{2}} \\ ⩽ C_{0} C_{2} C_{10} {[\int_{| {\hat{u}}_{n} | < C_{2} ‖ ϕ ‖_{\infty} ε^{- 1}} {(| {\hat{u}}_{n} {\tilde{ϕ}}_{n} |)}^{\frac{4}{2 + α}} d x]}^{\frac{2 + α}{4}} \\ ⩽ C_{0} C_{2} C_{10} ‖ {\hat{u}}_{n} ‖_{\frac{4}{α}} ‖ {\tilde{ϕ}}_{n} ‖_{2} ⩽ C_{11} ‖ {\tilde{ϕ}}_{n} ‖ \\ (4.44) & = o (1) . \end{aligned}$ Since $ε > 0$ is arbitrary, it follows from (4.43) and (4.44) that $\begin{array}{r} (4.45) & lim_{n \to \infty} \int_{R^{2}} (I_{β} * F ({\hat{u}}_{n})) f ({\hat{u}}_{n}) {\tilde{ϕ}}_{n} d x = 0 . \end{array}$ Similarly, one has $\begin{array}{r} (4.46) & lim_{n \to \infty} \int_{R^{2}} (I_{α} * F ({\hat{v}}_{n})) f ({\hat{v}}_{n}) {\tilde{ψ}}_{n} d x = 0 . \end{array}$ Then we deduce from (2.5), (4.38), (4.41), (4.42), (4.45), (4.46) and Lemma 4.6 that $\begin{array}{rcl} ⟨ Φ^{'} (\hat{u}, \hat{v}), (ϕ, ψ) ⟩ & = & \int_{R^{2}} [(\nabla \hat{u} \nabla ψ + \nabla \hat{v} \nabla ϕ) + V (x) (\hat{u} ψ + \hat{v} ϕ)] d x \\ - \int_{R^{2}} [(I_{α} * F (\hat{v})) f (\hat{v}) ψ + (I_{β} * F (\hat{u})) f (\hat{u}) ϕ] d x \\ = & lim_{n \to \infty} \int_{R^{2}} [(\nabla {\hat{u}}_{n} \nabla ψ + \nabla {\hat{v}}_{n} \nabla ϕ) + V (x) ({\hat{u}}_{n} ψ + {\hat{v}}_{n} ϕ)] d x \\ - lim_{n \to \infty} \int_{R^{2}} [(I_{α} * F ({\hat{v}}_{n})) f ({\hat{v}}_{n}) ψ + (I_{β} * F ({\hat{u}}_{n})) f ({\hat{u}}_{n}) ϕ] d x \\ = & lim_{n \to \infty} ⟨ Φ^{'} ({\hat{u}}_{n}, {\hat{v}}_{n}), (ϕ, ψ) ⟩ \\ = & lim_{n \to \infty} [⟨ Φ^{'} ({\hat{u}}_{n}, {\hat{v}}_{n}), (ϕ_{n}, ψ_{n}) ⟩ + ⟨ Φ^{'} ({\hat{u}}_{n}, {\hat{v}}_{n}), ({\tilde{ϕ}}_{n}, {\tilde{ψ}}_{n}) ⟩] \\ = & lim_{n \to \infty} ⟨ Φ^{'} ({\hat{u}}_{n}, {\hat{v}}_{n}), ({\tilde{ϕ}}_{n}, {\tilde{ψ}}_{n}) ⟩ \\ = & - lim_{n \to \infty} \int_{R^{2}} [(I_{α} * F ({\hat{v}}_{n})) f ({\hat{v}}_{n}) {\tilde{ψ}}_{n} + (I_{β} * F ({\hat{u}}_{n})) f ({\hat{u}}_{n}) {\tilde{ϕ}}_{n}] d x \\ = & 0 . \end{array}$ This shows that $(\hat{u}, \hat{v})$ is a nontrivial solution of system (1.1).

Since $\hat{z} = (\hat{u}, \hat{v}) \in M$ , one has $Φ (\hat{z}) ⩾ b$ . By $(F 2)$ , $(F 5)$ and Fatou’s lemma, one has $\begin{array}{rcl} b & ⩾ & \bar{c} = lim_{n \to \infty} [Φ ({\hat{z}}_{n}) - \frac{1}{2} ⟨ Φ^{'} ({\hat{z}}_{n}), {\hat{z}}_{n} ⟩] \\ ⩾ & \frac{1}{2} \underset{n \to \infty}{lim inf} \int_{R^{2}} [(I_{α} * F (v_{n})) (f (v_{n}) v_{n} - F (v_{n})) + (I_{β} * F (u_{n})) (f (u_{n}) u_{n} - F (u_{n}))] d x \\ ⩾ & \frac{1}{2} \int_{R^{2}} \underset{n \to \infty}{lim inf} [(I_{α} * F (v_{n})) (f (v_{n}) v_{n} - F (v_{n})) + (I_{β} * F (u_{n})) (f (u_{n}) u_{n} - F (u_{n}))] d x \\ = & \frac{1}{2} \int_{R^{2}} [(I_{α} * F (v)) (f (v) v - F (v)) + (I_{β} * F (u)) (f (u) u - F (u))] d x \\ = & Φ (\hat{z}) - \frac{1}{2} ⟨ Φ^{'} (\hat{z}), \hat{z} ⟩ \\ = & Φ (\hat{z}), \end{array}$ which together with (2.20) yields $Φ (\hat{z}) = b > 0$ . Thus $\hat{z} = (\hat{u}, \hat{v})$ is a ground states solution of system (1.1). We complete the proof. □

Footnotes

Acknowledgements

This work is partially supported by the National Natural Science Foundation of China (No. 12171486), the Fundamental Research Funds for the Central Universities of Central South University(No. 2024ZZTS0121), the Project for Young Backbone Teachers of Hunan Province (No. 10900-150220002) and Natural Science Foundation for Excellent Young Scholars of Hunan Province (No. 2023JJ20057).

References

Ackermann, On a periodic Schrödinger equation with nonlocal superlinear part, Math. Z. 248 (2004), 423–443.

Adachi and

Tanaka, Trudinger type inequalities in

R^{N}

and their best exponents, Proc. Amer. Math. Soc. 128 (2000), 2051–2057. doi:10.1090/S0002-9939-99-05180-1.

Aghajani and

Kinnunen, Supersolutions to nonautonomous Choquard equations in general domains, Adv. Nonlinear Anal. 12(1) (2023), 20230107, 21 pp.

F.S.B.

Albuquerque,

J.M.

do Ó and

E.S.

Medeiros, On a class of Hamiltonian elliptic systems involving unbounded or decaying potentials in dimension two, Math. Nachr. 289 (2016), 1568–1584. doi:10.1002/mana.201400203.

C.O.

Alves,

Cassani,

Tarsi and

M.B.

Yang, Existence and concentration of ground state solutions for a critical nonlocal Schrödinger equation in

R^{2}

, J. Differential Equations 261 (2016), 1933–1972. doi:10.1016/j.jde.2016.04.021.

C.O.

Alves and

G.F.

Germano, Ground state solution for a class of indefinite variational problems with critical growth, J. Differential Equations 265 (2018), 444–477. doi:10.1016/j.jde.2018.02.039.

Bartolo,

Benci and

Fortunato, Abstract critical point theorems and applications to some nonlinear problems with “strong” resonance at infinity, Nonlinear Anal. 7 (1983), 981–1012. doi:10.1016/0362-546X(83)90115-3.

Bonheure,

E.M.

dos Santos and

Tavares, Hamiltonian elliptic systems: A guide to variational frameworks, Port. Math. 71 (2014), 301–395. doi:10.4171/pm/1954.

D.M.

Cao, Nontrivial solution of semilinear elliptic equation with critical exponent in

R^{2}

, Comm. Partial Differential Equations 17 (1992), 407–435. doi:10.1080/03605309208820848.

10.

Cassani,

Sani and

Tarsi, Equivalent Moser type inequalities in

R^{2}

and the zero mass case, J. Funct. Anal. 267 (2014), 4236–4263. doi:10.1016/j.jfa.2014.09.022.

11.

S.T.

Chen,

D.D.

Qin,

V.D.

Radulescu and

X.H.

Tang, Ground states for quasilinear equations of N-Laplacian type with critical exponential growth and lack of compactness, Sci. China Math. (2024). doi:10.1007/s11425-023-2298-1.

12.

S.T.

Chen,

X.H.

Tang and

J.Y.

Wei, Nehari-type ground state solutions for a Choquard equation with doubly critical exponents, Adv. Nonlinear Anal. 10 (2021), 152–171. doi:10.1515/anona-2020-0118.

13.

D.G.

de Figueiredo,

J.M.

do Ó and

Ruf, Critical and subcritical elliptic systems in dimension two, Indiana Univ. Math. J. 53 (2004), 1037–1054. doi:10.1512/iumj.2004.53.2402.

14.

D.G.

de Figueiredo,

J.M.

do Ó and

J.J.

Zhang, Ground state solutions of Hamiltonian elliptic systems in dimension two, Proc. Roy. Soc. Edinburgh Sect. A 150 (2020), 1737–1768. doi:10.1017/prm.2018.78.

15.

D.G.

de Figueiredo,

O.H.

Miyagaki and

Ruf, Elliptic equations in

R^{2}

with nonlinearities in the critical growth range, Calc. Var. Partial Differential Equations 3 (1995), 139–153. doi:10.1007/BF01205003.

16.

de Souza and

J.M.

do Ó, Hamiltonian elliptic systems in

R^{2}

with subcritical and critical exponential growth, Ann. Mat. Pura Appl. 195 (2016), 935–956. doi:10.1007/s10231-015-0498-7.

17.

S.B.

Deng and

J.W.

Yu, On a class of singular Hamiltonian Choquard-type elliptic systems with critical exponential growth, J. Math. Phys. 63 (2022), 121501. doi:10.1063/5.0110352.

18.

S.B.

Deng and

J.W.

Yu, Existence of solution for a class of fractional Hamiltonian-type elliptic systems with exponential critical growth in

R

, J. Math. Phys. 65 (2024), 031502. doi:10.1063/5.0174408.

19.

Y.H.

Ding, Variational Methods for Strongly Indefinite Problems, World Scientific, 2007.

20.

Fröhlich, Theory of electrical breakdown in ionic crystal, Proc. Roy. Soc. Edinburgh Sect. A 160 (1937), 230–241.

21.

Ghimenti and

Van Schaftingen, Nodal solutions for the Choquard equation, J. Funct. Anal. 271 (2016), 107–135. doi:10.1016/j.jfa.2016.04.019.

22.

N.Y.

Lam and

G.Z.

Lu, Elliptic equations and systems with subcritical and critical exponential growth without the Ambrosetti–Rabinowitz condition, J. Geom. Anal. 24 (2014), 118–143. doi:10.1007/s12220-012-9330-4.

23.

Lan,

He and

Meng, Normalized solutions for a critical fractional Choquard equation with a nonlocal perturbation, Adv. Nonlinear Anal. 12(1) (2023), 20230112, 40 pp.

24.

Y.R.S.

Leuyacc and

S.H.M.

Soares, On a Hamiltonian system with critical exponential growth, Milan J. Math. 87 (2019), 105–140. doi:10.1007/s00032-019-00294-3.

25.

E.H.

Lieb, Existence and uniqueness of the minimizing solution of Choquard’s nonlinear equation, Studies in Appl. Math. 57 (1977), 93–105. doi:10.1002/sapm197757293.

26.

E.H.

Lieb and

Loss, Analysis, Graduate Studies in Mathematics, Vol. 14, American Mathematical Society, Providence, RI, 1997.

27.

Ma and

Zhao, Classification of positive solitary solutions of the nonlinear Choquard equation, Arch. Ration. Mech. Anal. 195 (2010), 455–467. doi:10.1007/s00205-008-0208-3.

28.

B.B.V.

Maia and

O.H.

Miyagaki, On a class of Hamiltonian Choquard-type elliptic systems, J. Math. Phys. 61 (2020), 011502. doi:10.1063/1.5134788.

29.

B.B.V.

Maia and

O.H.

Miyagaki, Existence and nonexistence results for a class of Hamiltonian Choquard-type elliptic systems with lower critical growth on

R^{2}

, Proc. Roy. Soc. Edinburgh Sect. A 152 (2022), 1383–1410. doi:10.1017/prm.2021.57.

30.

Mattner, Strict definiteness of integrals via complete monotonicity of derivatives, Trans. Amer. Math. Soc. 349 (1997), 3321–3342. doi:10.1090/S0002-9947-97-01966-1.

31.

Moroz and

Van Schaftingen, Semi-classical states for the Choquard equation, Calc. Var. Partial Differential Equations 52 (2015), 199–235. doi:10.1007/s00526-014-0709-x.

32.

Moroz and

Van Schaftingen, Existence of groundstates for a class of nonlinear Choquard equations, Trans. Amer. Math. Soc. 367 (2015), 6557–6579. doi:10.1090/S0002-9947-2014-06289-2.

33.

Moroz and

Van Schaftingen, A guide to the Choquard equation, J. Fixed Point Theory Appl. 19 (2017), 773–813. doi:10.1007/s11784-016-0373-1.

34.

Pankov, Periodic nonlinear Schrödinger equation with application to photonic crystals, Milan J. Math. 73 (2005), 259–287. doi:10.1007/s00032-005-0047-8.

35.

Pekar, Untersuchung über die Elektronentheorie der Kristalle, Akademie Verlag, Berlin, 1954.

36.

D.D.

Qin,

V.D.

Rădulescu and

X.H.

Tang, Ground states and geometrically distinct solutions for periodic Choquard–Pekar equations, J. Differential Equations 275 (2021), 652–683. doi:10.1016/j.jde.2020.11.021.

37.

D.D.

Qin and

X.H.

Tang, On the planar Choquard equation with indefinite potential and critical exponential growth, J. Differential Equations 285 (2021), 40–98. doi:10.1016/j.jde.2021.03.011.

38.

D.D.

Qin,

X.H.

Tang and

Zhang, Ground states for planar Hamiltonian elliptic systems with critical exponential growth, J. Differential Equations 308 (2022), 130–159. doi:10.1016/j.jde.2021.10.063.

39.

U.B.

Severo,

de Souza and

Menezes, Hamiltonian systems involving exponential growth in

R^{2}

with general nonlinearities, Rev. Real Acad. Cienc. Exactas Fis. Nat. Ser. A Mat. 118 (2024), 43. doi:10.1007/s13398-023-01542-3.

40.

Szulkin and

Weth, Ground state solutions for some indefinite variational problems, J. Funct. Anal. 257 (2009), 3802–3822. doi:10.1016/j.jfa.2009.09.013.

41.

X.H.

Tang, Non-Nehari manifold method for asymptotically periodic Schrödinger equations, Sci. China Math. 58 (2015), 715–728. doi:10.1007/s11425-014-4957-1.

42.

Van Schaftingen and

J.K.

Xia, Groundstates for a local nonlinear perturbation of the Choquard equations with lower critical exponent, J. Math. Anal. Appl. 464 (2018), 1184–1202. doi:10.1016/j.jmaa.2018.04.047.

43.

Willem, Minimax Theorems, Birkhäuser, Boston, 1996.

44.

J.K.

Xia and

Z.Q.

Wang, Saddle solutions for the Choquard equation, Calc. Var. Partial Differential Equations 58 (2019), 85. doi:10.1007/s00526-019-1546-8.

45.

Zhang,

Q.F.

Wu and

D.D.

Qin, Semiclassical solutions for Choquard equations with Berestycki–Lions type conditions, Nonlinear Anal. 188 (2019), 22–49. doi:10.1016/j.na.2019.05.016.