Ground states for a linearly coupled system of Schrödinger equations on R N

Abstract

We study the following class of linearly coupled Schrödinger elliptic systems $\begin{matrix} \{\begin{matrix} - Δ u + V_{1} (x) u = μ | u |^{p - 2} u + λ (x) v, & x \in R^{N}, \\ - Δ v + V_{2} (x) v = | v |^{q - 2} v + λ (x) u, & x \in R^{N}, \end{matrix} \end{matrix}$ where $N ⩾ 3$ , $2 < p ⩽ q ⩽ 2^{*} = 2 N / (N - 2)$ and $μ ⩾ 0$ . We consider nonnegative potentials periodic or asymptotically periodic which are related with the coupling term $λ (x)$ by the assumption $| λ (x) | ⩽ δ \sqrt{V_{1} (x) V_{2} (x)}$ , for some $0 < δ < 1$ . We deal with three cases: Firstly, we study the subcritical case, $2 < p ⩽ q < 2^{*}$ , and we prove the existence of positive ground state for all parameter $μ ⩾ 0$ . Secondly, we consider the critical case, $2 < p < q = 2^{*}$ , and we prove that there exists $μ_{0} > 0$ such that the coupled system possesses positive ground state solution for all $μ ⩾ μ_{0}$ . In these cases, we use a minimization method based on Nehari manifold. Finally, we consider the case $p = q = 2^{*}$ , and we prove that the coupled system has no positive solutions. For that matter, we use a Pohozaev identity type.

Keywords

Coupled systems nonlinear Schrödinger equations lack of compactness ground states

1. Introduction

We are interested in establish existence and nonexistence results for the following class of linearly coupled systems involving nonlinear Schrödinger equations $\begin{matrix} (1) & \{\begin{matrix} - Δ u + V_{1} (x) u = μ | u |^{p - 2} u + λ (x) v, & x \in R^{N}, \\ - Δ v + V_{2} (x) v = | v |^{q - 2} v + λ (x) u, & x \in R^{N}, \end{matrix} \end{matrix}$ where $N ⩾ 3$ , $2 < p ⩽ q ⩽ 2^{*} = 2 N / (N - 2)$ is the critical Sobolev exponent. Our main goal here is to prove the existence of positive ground states for the subcritical case, that is, when $2 < p ⩽ q < 2^{*}$ and for the critical case when $2 < p < q = 2^{*}$ . In the critical case, the existence of ground state will be related with the parameter μ introduced in the first equation. For the critical case when $p = q = 2^{*}$ , we make use of a Pohozaev type identity to prove that System (1) does not admit positive solution. We are concerned with two classes of nonnegative potentials: periodic and asymptotically periodic. Before we introduce our assumptions and the main results, we give a brief motivation to study this class of systems.

1.1. Motivation and related results

Solutions of System (1) are related with solutions of the following two-component system of nonlinear Schrödinger equations $\begin{matrix} (2) & \{\begin{matrix} - i \frac{\partial ψ}{\partial t} = Δ ψ - V_{1} (x) ψ + μ | ψ |^{p - 2} ψ + λ (x) ϕ, & x \in R^{N}, t ⩾ 0, \\ - i \frac{\partial ϕ}{\partial t} = Δ ϕ - V_{2} (x) ϕ + | ϕ |^{p - 2} ϕ + λ (x) ψ, & x \in R^{N}, t ⩾ 0 . \end{matrix} \end{matrix}$ Such class of systems arise in various branches of mathematical physics and nonlinear optics, see for instance [1]. For System (2), a solution of the form $\begin{matrix} (ψ (t, x), ϕ (t, x)) = (exp (- i E t) u (x), exp (- i E t) v (x)), \end{matrix}$ where E is some real constant, is called standing wave solution. Moreover, $(ψ, ϕ)$ is a solution of (2) if and only if $(u, v)$ solves the following system $\begin{matrix} \{\begin{matrix} - Δ u + (V_{1} (x) - E) u = μ | u |^{p - 2} u + λ (x) v, & x \in R^{N}, \\ - Δ v + (V_{2} (x) - E) v = | v |^{q - 2} v + λ (x) u, & x \in R^{N} . \end{matrix} \end{matrix}$ For convenience and without loss of generality, it is replaced $V_{i} (x) - E$ by $V_{i} (x)$ , that is, it is shifted E to 0. Thus, it turn to consider the coupled system (1).

When $λ (x) \equiv 0$ , $V_{1} (x) \equiv V_{2} (x) \equiv V (x)$ , $u (x) \equiv v (x)$ , $μ = 1$ and $p = q$ , System (1) reduces to the scalar equation $- Δ u + V (x) u = | u |^{p - 2} u$ , in $R^{N}$ . There are many papers that studied this class of Schrödinger equations under many different assumptions on the potential and nonlinearity. The literature is rather extensive, see for instance [3–6,10,18,19,22] and references therein.

Our work was inspired by some papers that have appeared in the recent years concerning the study of coupled systems involving nonlinear Schrödinger equations by using variational approach. In [8], Z. Chen and W. Zou studied the existence of ground states for the following class of critical coupled system with constant potentials $\begin{matrix} (3) & \{\begin{matrix} - Δ u + μ u = | u |^{p - 2} u + λ v, & x \in R^{N}, \\ - Δ v + ν v = | v |^{2^{*} - 2} v + λ u, & x \in R^{N} . \end{matrix} \end{matrix}$ They proved that there exists critical parameters $μ_{0} > 0$ and $λ_{μ, ν} \in [\sqrt{(μ - μ_{0}) ν}, \sqrt{μ ν})$ such that (3) has a positive ground state when $λ > λ_{μ, ν}$ and has no ground state solutions when $μ > μ_{0}$ and $λ < λ_{μ, ν}$ . In [7], the same authors studied a class of coupled systems involving general nonlinearities in the subcritical sense. In [12], Z. Guo and W. Zou obtained existence of positive ground states for another class of critical coupled systems. For more existence results concerning coupled systems we refer the readers to [2,14,16,17,21] and references therein.

Motivated by the above discussion, the current paper is concerned to study the class of coupled systems introduced by (1) in the subcritical and critical sense. This class of systems is characterized by its lack of compactness due to the fact that the equations are defined in whole Euclidean space $R^{N}$ , which roughly speaking, originates from the invariance of $R^{N}$ with respect to translation and dilation. Furthermore, we have the fact that (1) involves strongly coupled Schrödinger elliptic equations because of the linear terms in the right hand side. To overcome these difficulties, we shall use a variational approach based on Nehari manifold in combination with a lemma due to P.L. Lions (see Lemma 3.1).

1.2. Assumptions

Firstly, we deal with the following class of coupled systems $\begin{matrix} (S_{o}^{μ}) & \{\begin{matrix} - Δ u + V_{1, o} (x) u = μ | u |^{p - 2} u + λ_{o} (x) v, & x \in R^{N}, \\ - Δ v + V_{2, o} (x) v = | v |^{q - 2} v + λ_{o} (x) u, & x \in R^{N}, \end{matrix} \end{matrix}$ where $V_{1, o} (x)$ , $V_{2, o} (x)$ and $λ_{o} (x)$ denote periodic functions. In view of the presence of the potentials we introduce the following space $\begin{matrix} E_{i, o} = {u \in H^{1} (R^{N}) : \int_{R^{N}} V_{i, o} (x) u^{2} d x < + \infty}, i = 1, 2, \end{matrix}$ endowed with the inner product $\begin{matrix} {(u, v)}_{E_{i, o}} = \int_{R^{N}} \nabla u \nabla v d x + \int_{R^{N}} V_{i, o} (x) u v d x, \end{matrix}$ to which corresponds the induced norm $‖ u ‖_{E_{i, o}}^{2} = {(u, u)}_{E_{i, o}}$ . In order to establish a variational approach to treat System ( $S_{o}^{μ}$ ), we need to require suitable assumptions on the potentials. For each $i = 1, 2$ , we assume that

$V_{i, o}, λ_{o} \in C (R^{N})$ are 1-periodic in each of $x_{1}, x_{2}, \dots, x_{N}$ .

$V_{i, o} (x) ⩾ 0$ for all $x \in R^{N}$ and $\begin{matrix} ν_{i, o} = inf_{u \in E_{i, o}} {\int_{R^{N}} | \nabla u |^{2} d x + \int_{R^{N}} V_{i, o} (x) u^{2} d x : \int_{R^{N}} u^{2} d x = 1} > 0 . \end{matrix}$

$| λ_{o} (x) | ⩽ δ \sqrt{V_{1, o} (x) V_{2, o} (x)}$ , for some $δ \in (0, 1)$ , for all $x \in R^{N}$ .

$0 < λ_{o} (x) ⩽ δ \sqrt{V_{1, o} (x) V_{2, o} (x)}$ , for some $δ \in (0, 1)$ , for all $x \in R^{N}$ .

We set the product space $E_{o} = E_{1, o} \times E_{2, o}$ . We have that $E_{o}$ is a Hilbert space when endowed with the inner product $\begin{matrix} {((u, v), (w, z))}_{E_{o}} = {(u, w)}_{E_{1, o}} + {(v, z)}_{E_{2, o}}, \end{matrix}$ to which corresponds the induced norm $\begin{matrix} {‖ (u, v) ‖}_{E_{o}}^{2} = {((u, v), (u, v))}_{E_{o}} = ‖ u ‖_{E_{1, o}}^{2} + ‖ v ‖_{E_{2, o}}^{2} . \end{matrix}$ Associated to System ( $S_{o}^{μ}$ ) we have the functional $I_{μ, o} : E_{o} \to R$ defined by $\begin{matrix} I_{μ, o} (u, v) = \frac{1}{2} ({‖ (u, v) ‖}_{E_{o}}^{2} - 2 \int_{R^{N}} λ_{o} (x) u v d x) - \frac{μ}{p} ‖ u ‖_{p}^{p} - \frac{1}{q} ‖ v ‖_{q}^{q} . \end{matrix}$ Using our assumptions we can check that $I_{μ, o}$ is well defined and is of class $C^{2}$ with derivative given by $\begin{array}{l} ⟨ I_{μ, o}^{'} (u, v), (ϕ, ψ) ⟩ = {((u, v), (ϕ, ψ))}_{E_{o}} - \int_{R^{N}} (μ | u |^{p - 2} u ϕ + | v |^{q - 2} v ψ + λ_{o} (x) (u ψ + v ϕ)) d x, \end{array}$ where $(ϕ, ψ) \in C_{0}^{\infty} (R^{N}) \times C_{0}^{\infty} (R^{N})$ . Thus critical points of $I_{μ, o}$ correspond to weak solutions of ( $S_{o}^{μ}$ ) and conversely.

We say that a solution $(u_{0}, v_{0}) \in E_{o}$ for System ( $S_{o}^{μ}$ ) is a ground state (or least energy) solution if $(u_{0}, v_{0}) \neq (0, 0)$ and its energy is minimal among the energy of all nontrivial solutions, that is, $I_{μ, o} (u_{0}, v_{0}) ⩽ I_{μ, o} (u, v)$ for any other solution $(u, v) \in E_{o} ∖ {(0, 0)}$ . We say that $(u_{0}, v_{0})$ is nonnegative (nonpositive) if $u_{0}, v_{0} ⩾ 0$ ( $u_{0}, v_{0} ⩽ 0$ ) and positive (negative) if $u_{0}, v_{0} > 0$ ( $u_{0}, v_{0} < 0$ ) respectively.

We are also concerned with the existence of ground states for the following class of coupled systems $\begin{matrix} (S^{μ}) & \{\begin{matrix} - Δ u + V_{1} (x) u = μ | u |^{p - 2} u + λ (x) v, & x \in R^{N}, \\ - Δ v + V_{2} (x) v = | v |^{q - 2} v + λ (x) u, & x \in R^{N}, \end{matrix} \end{matrix}$ when the potentials $V_{1} (x)$ , $V_{2} (x)$ and $λ (x)$ are asymptotically periodic at infinity, that is, they are infinity limit of periodic functions $V_{1, o} (x)$ , $V_{2, o} (x)$ and $λ_{o} (x)$ . In analogous way, we may define the suitable product space $E = E_{1} \times E_{2}$ considering the asymptotically periodic potential $V_{i} (x)$ instead $V_{i, o} (x)$ . In order to give a variational approach for our problem, for $i = 1, 2$ we assume the following hypotheses:

$V_{i}, λ \in C (R^{N})$ , $V_{i} (x) < V_{i, o} (x)$ , $λ_{o} (x) < λ (x)$ , for all $x \in R^{N}$ and $\begin{matrix} lim_{| x | \to + \infty} | V_{i, o} (x) - V_{i} (x) | = 0 and lim_{| x | \to + \infty} | λ (x) - λ_{o} (x) | = 0 . \end{matrix}$

$V_{i} (x) ⩾ 0$ for all $x \in R^{N}$ and $\begin{matrix} ν_{i} = inf_{u \in E_{i}} {\int_{R^{N}} | \nabla u |^{2} d x + \int_{R^{N}} V_{i} (x) u^{2} d x : \int_{R^{N}} u^{2} d x = 1} > 0 . \end{matrix}$

$| λ (x) | ⩽ δ \sqrt{V_{1} (x) V_{2} (x)}$ , for some $δ \in (0, 1)$ , for all $x \in R^{N}$ .

$0 < λ (x) ⩽ δ \sqrt{V_{1} (x) V_{2} (x)}$ , for some $δ \in (0, 1)$ , for all $x \in R^{N}$ .

1.3. Statement of the main results

The main results of the paper are the following:

Theorem 1.1.
Assume that ( $V_{1}$ )–( $V_{3}$ ) hold. If $2 < p ⩽ q < 2^{}$ , then System (* $S_{o}^{μ}$ ) possesses a nonnegative ground state solution $(u_{0}, v_{0}) \in C_{loc}^{1, β} (R^{N}) \times C_{loc}^{1, β} (R^{N})$ for some $β \in (0, 1)$ , for all $μ ⩾ 0$ . If ( $V_{3}^{'}$ ) holds, then the ground state is positive.
Theorem 1.2.
Assume that ( $V_{1}$ )–( $V_{3}$ ) hold. If $2 < p < q = 2^{}$ , then there exists* $μ_{0} > 0$ such that System ( $S_{o}^{μ}$ ) possesses a nonnegative ground state solution $(u_{0}, v_{0}) \in E_{o}$ , for all $μ ⩾ μ_{0}$ . If ( $V_{3}^{'}$ ) holds, then the ground state is positive.
Theorem 1.3.
Suppose that assumptions ( $V_{1}$ )–( $V_{6}$ ) hold. If $2 < p ⩽ q < 2^{}$ , then System (* $S^{μ}$ ) possesses a nonnegative ground state solution $(u_{0}, v_{0}) \in C_{loc}^{1, β} (R^{N}) \times C_{loc}^{1, β} (R^{N})$ for some $β \in (0, 1)$ , for all $μ ⩾ 0$ . Moreover, if $2 < p < q = 2^{}$ , then there exists* $μ_{0} > 0$ such that System ( $S^{μ}$ ) possesses a nonnegative ground state solution for all $μ ⩾ μ_{0}$ . If ( $V_{6}^{'}$ ) holds, then the ground states are positive.
Theorem 1.4.
Suppose that $p = q = 2^{}$ and* ( $V_{6}$ ) holds. In addition, for $i = 1, 2$ we consider the following assumptions:
$V_{i} \in C^{1} (R^{N})$ is nonnegative and $0 ⩽ ⟨ \nabla V_{i} (x), x ⟩ ⩽ C V_{i} (x)$ .

$λ \in C^{1} (R^{N})$ , $| ⟨ \nabla λ (x), x ⟩ | ⩽ C | λ (x) |$ and $⟨ \nabla λ (x), x ⟩ ⩽ 0$ .
Then, System ( $S^{μ}$ ) has no positive classical solution for all $μ ⩾ 0$ .
Remark 1.5.
A typical example of functions satisfying ( $V_{7}$ ) and ( $V_{8}$ ) is $λ (x) = - (1 / 4) ‖ x ‖^{2}$ and $V_{i} (x) = (1 / 2) ‖ x ‖^{2}$ .

1.4. Notation

Let us introduce the following notation:

C, $\tilde{C}$ , $C_{1}$ , $C_{2}, \dots$ denote positive constants (possibly different).

$B_{R} (x_{0})$ denotes the open ball centered at $x_{0}$ and radius $R > 0$ .

The norm in $L^{p} (R^{N})$ and $L^{\infty} (R^{N})$ , will be denoted respectively by $‖ \cdot ‖_{p}$ and $‖ \cdot ‖_{\infty}$ .

$o_{n} (1)$ denotes a sequence which converges to 0 as $n \to \infty$ .

1.5. Outline

In the forthcoming section we introduce and give some properties of the Nehari manifold associated to ( $S_{o}^{μ}$ ). In Section 3, we deal with System ( $S_{o}^{μ}$ ) with subcritical growth: $2 < p ⩽ q < 2^{*}$ . For this matter we use a minimization method based on Nehari manifold to get a positive ground state solution and a bootstrap argument to obtain regularity. In Section 4, we study System ( $S_{o}^{μ}$ ) with critical growth, precisely: $2 < p < q = 2^{*}$ . In the periodic case, the key point is to use the invariance of the energy functional under translations to recover the compactness of the minimizing sequence. In Section 5, we study the existence of ground states when the potentials are asymptotically periodic. For this purpose, we establish a relation between the energy levels associated to Systems ( $S_{o}^{μ}$ ) and ( $S^{μ}$ ). In Section 6, we make use of Pohozaev type identity to prove the nonexistence of positive classical solutions for System ( $S^{μ}$ ) in the critical case, $p = q = 2^{*}$ .

2. Preliminary results

In this section we provide preliminary results used throughout the paper.

Lemma 2.1.
If ( $V_{3}$ ) holds, then we have $\begin{matrix} (4) & {‖ (u, v) ‖}_{E_{o}}^{2} - 2 \int_{R^{N}} λ_{o} (x) u v d x ⩾ (1 - δ) {‖ (u, v) ‖}_{E_{o}}^{2}, for all (u, v) \in E_{o} . \end{matrix}$
Proof.
For $(u, v) \in E_{o}$ , we have $\begin{matrix} 0 ⩽ {(\sqrt{V_{1, o} (x)} | u | - \sqrt{V_{2, o} (x)} | v |)}^{2} = V_{1, o} (x) u^{2} - 2 \sqrt{V_{1, o} (x)} | u | \sqrt{V_{2, o} (x)} | v | + V_{2, o} (x) v^{2}, \end{matrix}$ which together with assumption ( $V_{3}$ ) implies that $\begin{array}{rcl} - 2 \int_{R^{N}} λ_{o} (x) u v d x & ⩾ & - 2 δ \int_{R^{N}} \sqrt{V_{1, o} (x)} | u | \sqrt{V_{2, o} (x)} | v | d x \\ ⩾ & - δ (\int_{R^{N}} V_{1, o} (x) u^{2} d x + \int_{R^{N}} V_{2, o} (x) v^{2} d x) \\ ⩾ & - δ {‖ (u, v) ‖}_{E_{o}}^{2}, \end{array}$ which easily implies that (4) holds. □

In order to prove the existence of ground states, we introduce the Nehari manifold associated to System ( $S_{o}^{μ}$ ) $\begin{matrix} N_{μ, o} = {(u, v) \in E_{o} ∖ {(0, 0)} : ⟨ I_{μ, o}^{'} (u, v), (u, v) ⟩ = 0} . \end{matrix}$ Notice that if $(u, v) \in N_{μ, o}$ , then $\begin{matrix} (5) & {‖ (u, v) ‖}_{E_{o}}^{2} - 2 \int_{R^{N}} λ_{o} (x) u v d x = μ ‖ u ‖_{p}^{p} + ‖ v ‖_{q}^{q} . \end{matrix}$
Lemma 2.2.
There exists $α > 0$ such that $\begin{matrix} (6) & {‖ (u, v) ‖}_{E_{o}} ⩾ α, for all (u, v) \in N_{μ, o} . \end{matrix}$ Moreover, $N_{μ, o}$ is a $C^{1}$ -manifold.
Proof.
Let $(u, v) \in N_{μ, o}$ . By using (4), (5) and Sobolev embedding, we deduce that $\begin{matrix} (1 - δ) {‖ (u, v) ‖}_{E_{o}}^{2} ⩽ {‖ (u, v) ‖}_{E_{o}}^{2} - 2 \int_{R^{N}} λ_{o} (x) u v d x ⩽ C ({‖ (u, v) ‖}_{E_{o}}^{p} + {‖ (u, v) ‖}_{E_{o}}^{q}) . \end{matrix}$ Hence, we have that $\begin{matrix} 0 < \frac{1 - δ}{C} ⩽ {‖ (u, v) ‖}_{E_{o}}^{p - 2} + {‖ (u, v) ‖}_{E_{o}}^{q - 2}, \end{matrix}$ which implies that (6) holds. Now, let $J_{μ, o} : E_{o} ∖ {(0, 0)} \to R$ be the $C^{1}$ -functional defined by $\begin{matrix} J_{μ, o} (u, v) = ⟨ I_{μ, o}^{'} (u, v), (u, v) ⟩ = {‖ (u, v) ‖}_{E_{o}}^{2} - 2 \int_{R^{N}} λ_{o} (x) u v d x - μ ‖ u ‖_{p}^{p} - ‖ v ‖_{q}^{q} . \end{matrix}$ Notice that $N_{μ, o} = J_{μ, o}^{- 1} (0)$ . If $(u, v) \in N_{μ, o}$ , then it follows from (5) that $\begin{array}{rcl} ⟨ J_{μ, o}^{'} (u, v), (u, v) ⟩ & = & 2 ({‖ (u, v) ‖}_{E_{o}}^{2} - 2 \int_{R^{N}} λ_{o} (x) u v d x) - μ p ‖ u ‖_{p}^{p} - q ‖ v ‖_{q}^{q} \\ = & (2 - p) ({‖ (u, v) ‖}_{E_{o}}^{2} - 2 \int_{R^{N}} λ_{o} (x) u v d x) + (p - q) ‖ v ‖_{q}^{q}, \end{array}$ which together with (4), (6) and the fact that $2 < p ⩽ q$ implies that $\begin{array}{rcl} (7) & ⟨ J_{μ, o}^{'} (u, v), (u, v) ⟩ ⩽ (2 - p) (1 - δ) {‖ (u, v) ‖}_{E_{o}}^{2} ⩽ (2 - p) (1 - δ) α < 0 . \end{array}$ Thus, 0 is a regular value of $J_{μ, o}$ and therefore $N_{μ, o}$ is a $C^{1}$ -manifold. □
Remark 2.3.
If $(u_{0}, v_{0}) \in N_{μ, o}$ is a critical point of $I_{μ, o} |_{N_{μ, o}}$ , then $I_{μ, o}^{'} (u_{0}, v_{0}) = 0$ . In fact, notice that $I_{μ, o}^{'} (u_{0}, v_{0}) = η J_{μ, o}^{'} (u_{0}, v_{0})$ , where $η \in R$ is the corresponding Lagrange multiplier. Taking the scalar product with $(u_{0}, v_{0})$ and using (7) we conclude that $η = 0$ .
Lemma 2.4.
Assume ( $V_{3}$ ) holds. Thus, for any $(u, v) \in E_{o} ∖ {(0, 0)}$ , there exists a unique $t_{μ} > 0$ , depending on μ and $(u, v)$ , such that $\begin{matrix} (t_{μ} u, t_{μ} v) \in N_{μ, o} and I_{μ, o} (t_{μ} u, t_{μ} v) = max_{t ⩾ 0} I_{μ, o} (t u, t v) . \end{matrix}$
Proof.
Let $(u, v) \in E_{o} ∖ {(0, 0)}$ be fixed and consider the function $g : [0, \infty) \to R$ defined by $g (t) = I_{μ, o} (t u, t v)$ . Notice that $⟨ I_{μ, o}^{'} (t u, t v), (t u, t v) ⟩ = t g^{'} (t)$ . Therefore, $t_{μ}$ is a positive critical point of g if and only if $(t_{μ} u, t_{μ} v) \in N_{μ, o}$ . It follows from assumption ( $V_{3}$ ) that $\begin{matrix} {‖ (u, v) ‖}_{E_{o}}^{2} - 2 \int_{R^{N}} λ_{o} (x) u v d x ⩾ 0, for all (u, v) \in E_{o} . \end{matrix}$ Since $2 < p ⩽ q$ and $\begin{matrix} g (t) = \frac{t^{2}}{2} ({‖ (u, v) ‖}_{E_{o}}^{2} - 2 \int_{R^{N}} λ_{o} (x) u v d x) - \frac{t^{p}}{p} μ ‖ u ‖_{p}^{p} - \frac{t^{q}}{q} ‖ v ‖_{q}^{q}, \end{matrix}$ we conclude that $g (t) < 0$ for $t > 0$ sufficiently large. On the other hand, by using ( $V_{3}$ ) and Sobolev embeddings, we have that $\begin{array}{rcl} g (t) & ⩾ & (1 - δ) \frac{t^{2}}{2} {‖ (u, v) ‖}_{E_{o}}^{2} - C_{1} μ \frac{t^{p}}{p} ‖ u ‖_{E_{1, o}}^{p} - C_{2} \frac{t^{q}}{q} ‖ v ‖_{E_{2, o}}^{q} \\ ⩾ & t^{2} {‖ (u, v) ‖}_{E_{o}}^{2} (\frac{1 - δ}{2} - C_{1} μ \frac{t^{p - 2}}{p} {‖ (u, v) ‖}_{E_{o}}^{p - 2} - C_{2} \frac{t^{q - 2}}{q} {‖ (u, v) ‖}_{E_{o}}^{q - 2}) > 0, \end{array}$ provided $t > 0$ is sufficiently small. Thus g has maximum points in $(0, \infty)$ . Suppose that there exists $t_{1}, t_{2} > 0$ with $t_{1} < t_{2}$ such that $g^{'} (t_{1}) = g^{'} (t_{2}) = 0$ . Since every critical point of g satisfies $\begin{matrix} (8) & {‖ (u, v) ‖}_{E_{o}}^{2} - 2 \int_{R^{N}} λ_{o} (x) u v d x = t^{p - 2} μ ‖ u ‖_{p}^{p} + t^{q - 2} ‖ v ‖_{q}^{q}, \end{matrix}$ we have that $(t_{1}^{p - 2} - t_{2}^{p - 2}) μ ‖ u ‖_{p}^{p} + (t_{1}^{q - 2} - t_{2}^{q - 2}) ‖ v ‖_{q}^{q} = 0$ . Thus $u = v = 0$ which is impossible and the proof is complete. □

Let us define the Nehari energy level associated with System ( $S_{o}^{μ}$ ) $\begin{matrix} c_{N_{μ, o}} = inf_{(u, v) \in N_{μ, o}} I_{μ, o} (u, v) . \end{matrix}$ We claim that $c_{N_{μ, o}}$ is positive. In fact, for any $(u, v) \in N_{μ, o}$ we can deduce that $\begin{matrix} I_{μ, o} (u, v) = (\frac{1}{2} - \frac{1}{p}) ({‖ (u, v) ‖}_{E_{o}}^{2} - 2 \int_{R^{N}} λ_{o} (x) u v d x) + (\frac{1}{p} - \frac{1}{q}) ‖ v ‖_{q}^{q} . \end{matrix}$ Since $2 < p ⩽ q$ , it follows from (4) and (6) that $\begin{matrix} I_{μ, o} (u, v) ⩾ (\frac{1}{2} - \frac{1}{p}) (1 - δ) {‖ (u, v) ‖}_{E_{o}}^{2} ⩾ (\frac{1}{2} - \frac{1}{p}) (1 - δ) α > 0 . \end{matrix}$ Remark 2.5.
Although we used the notation for periodic functions, all results of this section remain true for asymptotically periodic functions.

3. Proof of Theorem 1.1

We can use Ekeland’s variational principle (see [9]) to obtain a sequence ${(u_{n}, v_{n})}_{n} \subset N_{μ, o}$ such that $\begin{matrix} (9) & I_{μ, o} (u_{n}, v_{n}) \to c_{N_{μ, o}} and I_{μ, o}^{'} (u_{n}, v_{n}) \to 0 . \end{matrix}$ Notice that ${(u_{n}, v_{n})}_{n}$ is bounded. In fact, recalling that $p ⩽ q$ it follows from (4) and (5) that $\begin{array}{rcl} I_{μ, o} (u_{n}, v_{n}) & = & (\frac{1}{2} - \frac{1}{p}) ({‖ (u_{n}, v_{n}) ‖}_{E_{o}}^{2} - 2 \int_{R^{N}} λ_{o} (x) u_{n} v_{n} d x) + (\frac{1}{p} - \frac{1}{q}) ‖ v_{n} ‖_{q}^{q} \\ ⩾ & (\frac{1}{2} - \frac{1}{q}) (1 - δ) {‖ (u_{n}, v_{n}) ‖}_{E_{o}}^{2} . \end{array}$ Since $I_{μ, o} (u_{n}, v_{n})$ is bounded, we conclude that ${(u_{n}, v_{n})}_{n}$ is bounded in $E_{o}$ . Passing ${(u_{n}, v_{n})}_{n}$ to a subsequence, we way assume that $(u_{n}, v_{n}) ⇀ (u_{0}, v_{0})$ weakly in $E_{o}$ . By a standard argument, we have that $I_{μ, o}^{'} (u_{0}, v_{0}) = 0$ . We recall the following result due to P.L. Lions [20, Lemma 1.21] (see also [15]).

Lemma 3.1.
Let $r > 0$ and $2 ⩽ s < 2^{}$ . If* ${(u_{n})}_{n} \subset H^{1} (R^{N})$ is a bounded sequence such that $\begin{matrix} lim_{n \to + \infty} sup_{y \in R^{N}} \int_{B_{r} (y)} | u_{n} |^{s} d x = 0, \end{matrix}$ then $u_{n} \to 0$ in $L^{s} (R^{N})$ for $2 < s < 2^{}$ .
Proposition 3.2.
There exists a ground state solution for System (* $S_{o}^{μ}$ ).
Proof.
We split the argument into two cases.

Case 1. $(u_{0}, v_{0}) \neq (0, 0)$ .

In this case, $(u_{0}, v_{0})$ is a nontrivial critical point of the energy functional $I_{μ, o}$ . Thus, $(u_{0}, v_{0}) \in N_{μ, o}$ . It remains to prove that $I_{μ, o} (u_{0}, v_{0}) = c_{N_{μ, o}}$ . It is clear that $c_{N_{μ, o}} ⩽ I_{μ, o} (u_{0}, v_{0})$ . On the other hand, by using the semicontinuity of norm, we can deduce that $\begin{array}{rcl} c_{N_{μ, o}} + o_{n} (1) & = & I_{μ, o} (u_{n}, v_{n}) - \frac{1}{2} ⟨ I_{μ, o}^{'} (u_{n}, v_{n}), (u_{n}, v_{n}) ⟩ \\ = & (\frac{1}{2} - \frac{1}{p}) μ ‖ u_{n} ‖_{p}^{p} + (\frac{1}{2} - \frac{1}{q}) ‖ v_{n} ‖_{q}^{q} \\ ⩾ & (\frac{1}{2} - \frac{1}{p}) μ ‖ u_{0} ‖_{p}^{p} + (\frac{1}{2} - \frac{1}{q}) ‖ v_{0} ‖_{q}^{q} + o_{n} (1) \\ = & I_{μ, o} (u_{0}, v_{0}) - \frac{1}{2} ⟨ I_{μ, o}^{'} (u_{0}, v_{0}), (u_{0}, v_{0}) ⟩ + o_{n} (1) \\ = & I_{μ, o} (u_{0}, v_{0}) + o_{n} (1), \end{array}$ which implies that $c_{N_{μ, o}} ⩾ I_{μ, o} (u_{0}, v_{0})$ . Therefore, $I_{μ, o} (u_{0}, v_{0}) = c_{N_{μ, o}}$ .

Case 2. $(u_{0}, v_{0}) = (0, 0)$ .

We claim that there exist a sequence ${(y_{n})}_{n} \subset R^{N}$ and constants $R, ξ > 0$ such that $\begin{matrix} (10) & \underset{n \to \infty}{lim inf} \int_{B_{R} (y_{n})} (u_{n}^{2} + v_{n}^{2}) d x ⩾ ξ > 0 . \end{matrix}$ Suppose by contradiction that (10) does not hold. Thus, for any $R > 0$ we have $\begin{matrix} lim_{n \to \infty} sup_{y \in R^{N}} \int_{B_{R} (y)} (u_{n}^{2} + v_{n}^{2}) d x = 0 . \end{matrix}$ It follows from Lemma 3.1 that $u_{n} \to 0$ strongly in $L^{p} (R^{N})$ and $v_{n} \to 0$ strongly $L^{q} (R^{N})$ , for $2 < p, q < 2^{}$ . Since ${(u_{n}, v_{n})}_{n} \subset N_{μ, o}$ , we can deduce that $\begin{matrix} 0 < (1 - δ) α ⩽ (1 - δ) {‖ (u_{n}, v_{n}) ‖}_{E_{o}}^{2} ⩽ μ ‖ u_{n} ‖_{p}^{p} + ‖ v_{n} ‖_{q}^{q} \to 0, \end{matrix}$ which implies that $(u_{n}, v_{n}) \to 0$ strongly in $E_{o}$ . But this is impossible, since $I_{μ, o}$ is continuous and $I_{μ, o} (u_{n}, v_{n}) \to c_{N_{μ, o}} > 0$ . Therefore, (10) holds.

We may assume without loss of generality that ${(y_{n})}_{n} \subset Z^{N}$ . Let us consider the shift sequence $({\tilde{u}}_{n} (x), {\tilde{v}}_{n} (x)) = (u_{n} (x + y_{n}), v_{n} (x + y_{n}))$ . Since $V_{1, o} (\cdot)$ , $V_{2, o} (\cdot)$ and $λ_{o} (\cdot)$ are 1-periodic functions, it follows that the energy functional $I_{μ, o}$ is invariant under translations of the form $(u, v) \mapsto (u (\cdot - z), v (\cdot - z))$ with $z \in Z^{N}$ . By a careful computation we can deduce that $\begin{matrix} {‖ (u_{n}, v_{n}) ‖}_{E_{o}} = {‖ ({\tilde{u}}_{n}, {\tilde{v}}_{n}) ‖}_{E_{o}}, I_{μ, o} (u_{n}, v_{n}) = I_{μ, o} ({\tilde{u}}_{n}, {\tilde{v}}_{n}) \to c_{N_{μ, o}} and I_{μ, o}^{'} ({\tilde{u}}_{n}, {\tilde{v}}_{n}) \to 0 . \end{matrix}$ Moreover, arguing as before, we can conclude that ${({\tilde{u}}_{n}, {\tilde{v}}_{n})}_{n}$ is a bounded sequence in $E_{o}$ . In this way, there exists a critical point $(\tilde{u}, \tilde{v})$ of $I_{μ, o}$ , such that, up to a subsequence, $({\tilde{u}}_{n}, {\tilde{v}}_{n}) ⇀ (\tilde{u}, \tilde{v})$ weakly in $E_{o}$ and $({\tilde{u}}_{n}, {\tilde{v}}_{n}) \to (\tilde{u}, \tilde{v})$ strongly in $L^{2} (B_{R} (0)) \times L^{2} (B_{R} (0))$ . Thus, using (10) we obtain $\begin{matrix} \int_{B_{R} (0)} ({\tilde{u}}^{2} + {\tilde{v}}^{2}) d x = \underset{n \to \infty}{lim inf} \int_{B_{R} (0)} ({\tilde{u}}_{n}^{2} + {\tilde{v}}_{n}^{2}) d x = \underset{n \to \infty}{lim inf} \int_{B_{R} (y_{n})} (u_{n}^{2} + v_{n}^{2}) d x ⩾ ξ > 0 . \end{matrix}$ Therefore, $\tilde{u} ≢ 0$ or $\tilde{v} ≢ 0$ . The conclusion follows as in the Case* 1. □
Proposition 3.3.
There exists a nonnegative ground state solution $(\tilde{u}, \tilde{v}) \in C_{loc}^{1, β} (R^{N}) \times C_{loc}^{1, β} (R^{N})$ for System ( $S_{o}^{μ}$ ), for some $β \in (0, 1)$ .
Proof.
Let $(u_{0}, v_{0}) \in N_{μ, o}$ be the ground state obtained in the Proposition 3.2. From Lemma 2.4, there exists $t_{μ} > 0$ such that $(t_{μ} | u_{0} |, t_{μ} | v_{0} |) \in N_{μ, o}$ . Thus, we have that $\begin{matrix} I_{μ, o} (t_{μ} | u_{0} |, t_{μ} | v_{0} |) ⩽ I_{μ, o} (t_{μ} u_{0}, t_{μ} v_{0}) ⩽ max_{t ⩾ 0} I_{μ, o} (t u_{0}, t v_{0}) = I_{μ, o} (u_{0}, v_{0}) = c_{N_{μ, o}}, \end{matrix}$ which implies that $(t_{μ} | u_{0} |, t_{μ} | v_{0} |)$ is also a minimizer of $I_{μ, o}$ on $N_{μ, o}$ . Therefore, $(t_{μ} | u_{0} |, t_{μ} | v_{0} |)$ is a nonnegative ground state solution for System ( $S_{o}^{μ}$ ).

To prove the regularity, we use the standard bootstrap argument. We denote $(\tilde{u}, \tilde{v}) = (t_{μ} | u_{0} |, t_{μ} | v_{0} |)$ and we define $\begin{matrix} p_{1} (x) = μ | \tilde{u} |^{p - 2} \tilde{u} + λ_{o} (x) \tilde{v} - V_{1, o} (x) \tilde{u} and p_{2} (x) = | \tilde{v} |^{q - 2} \tilde{v} + λ_{o} (x) \tilde{u} - V_{2, o} (x) \tilde{v} . \end{matrix}$ Thus, $(\tilde{u}, \tilde{v})$ is a weak solution of the restricted problem $\begin{matrix} (11) & \{\begin{matrix} - Δ \tilde{u} = p_{1} (x), & x \in B_{1} (0), \\ - Δ \tilde{v} = p_{2} (x), & x \in B_{1} (0) . \end{matrix} \end{matrix}$ Using Sobolev embedding we have that $V_{1, o} \tilde{u}, V_{2, o} \tilde{v}, λ_{o} \tilde{u}, λ_{o} \tilde{v} \in L^{2^{}} (B_{1} (0))$ . Moreover, $| \tilde{u} |^{p - 2} \tilde{u} \in L^{r} (B_{1} (0))$ for all $1 ⩽ r ⩽ 2^{} / (p - 1)$ and $| \tilde{v} |^{q - 2} \tilde{v} \in L^{s} (B_{1} (0))$ for all $1 ⩽ s ⩽ 2^{} / (q - 1)$ . Let us define $r_{1} = 2^{} / (q - 1)$ . Since $p ⩽ q$ , it follows that $r_{1} ⩽ 2^{} / (p - 1)$ . Hence $| \tilde{u} |^{p - 2} \tilde{u} \in L^{r_{1}} (B_{1} (0))$ . Therefore, $p_{1}, p_{2} \in L^{r_{1}} (B_{1} (0))$ . On the other hand, for each $i = 1, 2$ let $w_{i}$ be the Newtonian potential of $p_{i} (x)$ . Thus, in light of [11, Theorem 9.9] we have $w_{i} \in W^{2, r_{1}} (B_{1} (0))$ and $\begin{matrix} (12) & \{\begin{matrix} Δ w_{1} = p_{1} (x), & x \in B_{1} (0), \\ Δ w_{2} = p_{2} (x), & x \in B_{1} (0) . \end{matrix} \end{matrix}$ Therefore, $(\tilde{u} - w_{1}, \tilde{v} - w_{2}) \in H^{1} (B_{1} (0)) \times H^{1} (B_{1} (0))$ is a weak solution of the problem $\begin{matrix} \{\begin{matrix} Δ z_{1} = 0, & in B_{1} (0), \\ Δ z_{2} = 0, & in B_{1} (0) . \end{matrix} \end{matrix}$ In view of [13, Corollary 1.2.1], we have that $(\tilde{u} - w_{1}, \tilde{v} - w_{2}) \in C^{\infty} (B_{1} (0)) \times C^{\infty} (B_{1} (0))$ . Therefore, $(\tilde{u}, \tilde{v}) \in W^{2, r_{1}} (B_{1} (0)) \times W^{2, r_{1}} (B_{1} (0))$ . Since $q - 1 < 2^{} - 1$ , there exists $δ > 0$ such that $(q - 1) (1 + δ) = 2^{} - 1$ . Thus, one has $\begin{matrix} (13) & r_{1} = \frac{2^{}}{q - 1} = 2^{} \frac{(1 + δ)}{2^{} - 1} = \frac{2 N}{N + 2} (1 + δ) . \end{matrix}$ Recall the Sobolev embedding $W^{2, r_{1}} (B_{1} (0)) ↪ L^{s_{1}} (B_{1} (0))$ with $s_{1} = N r_{1} / (N - 2 r_{1})$ . We claim that there exists $r_{2} \in (r_{1}, s_{1})$ such that $(\tilde{u}, \tilde{v}) \in W^{2, r_{2}} (B_{1} (0)) \times W^{2, r_{2}} (B_{1} (0))$ . Indeed, we define $r_{2} = s_{1} / (q - 1)$ and we note that $r_{2} < s_{1}$ . By using (13) we deduce that $\begin{matrix} \frac{r_{2}}{r_{1}} = \frac{N r_{1}}{(q - 1) (N - 2 r_{1}) r_{1}} = \frac{(N - 2) (1 + δ)}{N - 2 - 4 δ} > 1 + δ, \end{matrix}$ which implies that $r_{2} \in (r_{1}, s_{1})$ . By Sobolev embedding, we have $\begin{matrix} W^{2, r_{1}} (B_{1} (0)) ↪ L^{s_{1}} (B_{1} (0)) ↪ L^{r_{2}} (B_{1} (0)) . \end{matrix}$ Hence, $p_{1} (x), p_{2} (x) \in L^{r_{2}} (B_{1} (0))$ . From the same argument used before, we can conclude that $(\tilde{u}, \tilde{v}) \in W^{2, r_{2}} (B_{1} (0)) \times W^{2, r_{2}} (B_{1} (0))$ . Iterating, we obtain the following sequence $\begin{matrix} r_{n + 1} = \frac{1}{q - 1} (\frac{N r_{n}}{N - 2 r_{n}}) . \end{matrix}$ Notice that $r_{n + 1} \to \infty$ , as $n \to \infty$ . Therefore, $\begin{matrix} (\tilde{u}, \tilde{v}) \in W_{loc}^{2, r} (R^{N}) \times W_{loc}^{2, r} (R^{N}), for all 2 ⩽ r < \infty . \end{matrix}$ From Sobolev embedding, we have that $(\tilde{u}, \tilde{v}) \in C^{1, β} (B_{1} (0)) \times C^{1, β} (B_{1} (0))$ , for some $β \in (0, 1)$ . □
Proposition 3.4.
If ( $V_{3}^{'}$ ) holds, then the ground state is positive.
Proof.
Let $(\tilde{u}, \tilde{v}) \in E_{o} ∖ {(0, 0)}$ be the nonnegative ground state obtained in Proposition 3.3. Since $(\tilde{u}, \tilde{v}) \neq (0, 0)$ we may assume without loss of generality that $\tilde{u} \neq 0$ . We claim that $\tilde{v} \neq 0$ . In fact, arguing by contradiction we suppose that $\tilde{v} = 0$ . Thus, $\begin{matrix} 0 = ⟨ I_{μ, o}^{'} (\tilde{u}, 0), (0, ψ) ⟩ = - \int_{R^{N}} λ_{o} (x) \tilde{u} ψ d x, for all ψ \in C_{0}^{\infty} (R^{N}) . \end{matrix}$ Since $λ_{o} (x)$ is positive, we have that $\tilde{u} = 0$ which is a contradiction. Therefore, $\tilde{v} \neq 0$ .

Taking $(φ, 0)$ as test function one sees that $\begin{matrix} \int_{R^{N}} \nabla \tilde{u} \nabla φ d x + \int_{R^{N}} V_{1, o} (x) \tilde{u} φ d x = μ \int_{R^{N}} | \tilde{u} |^{p - 2} \tilde{u} φ d x + \int_{R^{N}} λ_{o} (x) \tilde{v} φ d x ⩾ 0, \end{matrix}$ for all $φ ⩾ 0$ , $φ \in C_{0}^{\infty} (R^{N})$ . Thus, we can deduce that $\begin{matrix} \int_{R^{N}} \nabla (- \tilde{u}) \nabla φ d x - \int_{R^{N}} [- V_{1, o} (x)] (- \tilde{u}) φ d x ⩽ 0, for all φ ⩾ 0, φ \in C_{0}^{\infty} (R^{N}) . \end{matrix}$ Moreover, since $V_{1, o} (x) ⩾ 0$ for all $x \in R^{N}$ , it follows that $\begin{matrix} - \int_{R^{N}} V_{1, o} (x) φ d x ⩽ 0, for all φ ⩾ 0, φ \in C_{0}^{\infty} (R^{N}) . \end{matrix}$ In order to prove that $(\tilde{u}, \tilde{v})$ is positive, we suppose by contradiction that there exists $p \in R^{N}$ such that $\tilde{u} (p) = 0$ . Thus, since $- \tilde{u} ⩽ 0$ in $R^{N}$ , for any $R > > R_{0} > 0$ we have that $\begin{matrix} 0 = sup_{B_{R_{0}} (p)} (- \tilde{u}) = sup_{B_{R} (p)} (- \tilde{u}) . \end{matrix}$ By the Strong Maximum Principle [11, Theorem 8.19] we conclude that $- \tilde{u} \equiv 0$ in $B_{R} (p)$ , for all $R > R_{0}$ . Therefore, $\tilde{u} \equiv 0$ in $R^{N}$ which is a contradiction. Therefore $\tilde{u} > 0$ in $R^{N}$ . Analogously we can prove that $\tilde{v} > 0$ in $R^{N}$ . Therefore, the ground state $(\tilde{u}, \tilde{v})$ is positive. □

Theorem 1.1 follows from Propositions 3.2, 3.3 and 3.4.
4. Proof of Theorem 1.2

In this section, we deal with System ( $S_{o}^{μ}$ ) when $2 < p < q = 2^{*}$ . Analogously to Theorem 1.1, we have a sequence ${(u_{n}, v_{n})}_{n} \subset N$ satisfying (9). Moreover, the sequence is bounded and $(u_{n}, v_{n}) ⇀ (u_{0}, v_{0})$ weakly in E. We have also that $(u_{0}, v_{0})$ is a critical point of the energy functional I. We denote by S the sharp constant of the embedding $D^{1, 2} (R^{N}) ↪ L^{2^{*}} (R^{N})$ $\begin{matrix} (14) & S {(\int_{R^{N}} | u |^{2^{*}} d x)}^{2 / 2^{*}} ⩽ \int_{R^{N}} | \nabla u |^{2} d x, \end{matrix}$ where $D^{1, 2} (R^{N}) : = {u \in L^{2^{*}} (R^{N}) : | \nabla u | \in L^{2} (R^{N})}$ . In order to get a nontrivial critical point for $I_{μ, o}$ we need the following lemma:

Lemma 4.1.
There exists $μ_{0} > 0$ such that $c_{N_{μ, o}} < \frac{1}{N} S^{N / 2}$ , for all $μ ⩾ μ_{0}$ .
Proof.
Let us consider $(u, v) \in E_{o}$ such that $u, v ⩾ 0$ and $u, v ≢ 0$ . It follows from Lemma 2.4 that there exists a unique $t_{μ} > 0$ , depending on $μ > 0$ and $(u, v)$ , such that $(t_{μ} u, t_{μ} v) \in N_{μ, o}$ . Thus, by using relation (8) we can conclude that $t_{μ} \to 0$ as $μ \to + \infty$ . Moreover, we have that $\begin{matrix} c_{N_{μ, o}} ⩽ I_{μ, o} (t_{μ} u, t_{μ} v) ⩽ \frac{t_{μ}^{2}}{2} ({‖ (u, v) ‖}_{E_{o}}^{2} - 2 \int_{R^{N}} λ_{o} (x) u v d x), \end{matrix}$ and the right hand side goes to zero as μ goes to infinity. Therefore, there exists $μ_{0} > 0$ such that $c_{N_{μ, o}} < \frac{1}{N} S^{N / 2}$ , for all $μ ⩾ μ_{0}$ . □

In analogous way to the proof of Theorem 1.1, we split the proof into two cases.

Case 1. $(u_{0}, v_{0}) \neq (0, 0)$ .

This case is completely similar to the proof of the subcritical case.

Case 2. $(u_{0}, v_{0}) = (0, 0)$ .

Let $μ_{0} > 0$ be the parameter obtained in the Lemma 4.1. We claim that if $μ ⩾ μ_{0}$ , then there exists a sequence ${(y_{n})}_{n} \subset R^{N}$ and constants $R, ξ > 0$ such that $\begin{matrix} (15) & \underset{n \to \infty}{lim inf} \int_{B_{R} (y_{n})} (u_{n}^{2} + v_{n}^{2}) d x ⩾ ξ > 0 . \end{matrix}$ In fact, suppose that (15) does not hold. Thus, for any $R > 0$ we have $\begin{matrix} lim_{n \to \infty} sup_{y \in R^{N}} \int_{B_{R} (y)} (u_{n}^{2} + v_{n}^{2}) d x = 0 . \end{matrix}$ It follows from Lemma 3.1 that $u_{n} \to 0$ strongly in $L^{p} (R^{N})$ , for $2 < p < 2^{}$ . Notice that $\begin{matrix} I_{μ, o} (u_{n}, v_{n}) - \frac{1}{2} ⟨ I_{μ, o}^{'} (u_{n}, v_{n}), (u_{n}, v_{n}) ⟩ = \frac{p - 2}{2 p} μ ‖ u_{n} ‖_{p}^{p} + \frac{1}{N} ‖ v_{n} ‖_{2^{}}^{2^{}}, \end{matrix}$ which together with (9) and Lemma 3.1 implies that $\begin{matrix} N c_{N_{μ, o}} + o_{n} (1) = N (I_{μ, o} (u_{n}, v_{n}) - \frac{1}{2} ⟨ I_{μ, o}^{'} (u_{n}, v_{n}), (u_{n}, v_{n}) ⟩ - \frac{p - 2}{2 p} μ ‖ u_{n} ‖_{p}^{p}) = ‖ v_{n} ‖_{2^{}}^{2^{}} . \end{matrix}$ Moreover, we can deduce that $\begin{matrix} N c_{N_{μ, o}} + o_{n} (1) = ‖ v_{n} ‖_{2^{}}^{2^{}} + μ ‖ u_{n} ‖_{p}^{p} + ⟨ I_{μ, o}^{'} (u_{n}, v_{n}), (u_{n}, v_{n}) ⟩ = {‖ (u_{n}, v_{n}) ‖}_{E_{o}}^{2} - 2 \int_{R^{N}} λ_{o} (x) u_{n} v_{n} d x . \end{matrix}$ The preceding computations implies that $\begin{matrix} N c_{N_{μ, o}} + o_{n} (1) = ‖ v_{n} ‖_{2^{}}^{2^{}} ⩽ S^{- \frac{N}{N - 2}} ‖ \nabla v_{n} ‖_{2}^{\frac{2 N}{N - 2}} ⩽ S^{- \frac{N}{N - 2}} {({‖ (u_{n}, v_{n}) ‖}_{E_{o}}^{2} - 2 \int_{R^{N}} λ_{o} (x) u_{n} v_{n} d x)}^{\frac{N}{N - 2}} . \end{matrix}$ Thus, we can conclude that $\begin{matrix} N c_{N_{μ, o}} + o_{n} (1) ⩽ {(\frac{N c_{N_{μ, o}}}{S})}^{\frac{N}{N - 2}} + o_{n} (1) . \end{matrix}$ Therefore, $c_{N_{μ, o}} ⩾ \frac{1}{N} S^{N / 2}$ , contradicting Lemma 4.1.

Since (15) holds, we can consider the shift sequence $({\tilde{u}}_{n} (x), {\tilde{v}}_{n} (x)) = (u_{n} (x + y_{n}), v_{n} (x + y_{n}))$ and we can repeat the same arguments used in the proof of Theorem 1.1 to finish the proof. Remark 4.2.
Let us set $Λ : = {μ > 0 : (S_{o}^{μ}) has ground state}$ . We have proved in Theorem 1.2 that Λ is nonempty. Naturally arise the following questions: $\tilde{μ} : = inf Λ > 0$ ? Λ is an interval? Can we use the approach to study the existence of ground states for the system of the form: $\begin{matrix} (S_{μ}) & \{\begin{matrix} - Δ u + V_{1, o} (x) u = | u |^{p - 2} u + λ (x) v, & x \in R^{N}, \\ - Δ v + V_{2, o} (x) v = μ | v |^{2^{} - 2} v + λ (x) u, & x \in R^{N} . \end{matrix} \end{matrix}$ Does System ( $S_{μ}$ ) possesses ground state solution for any $μ > 0$ ?

5. Proof of Theorem 1.3

In this section we will be concerned with the existence of ground states for the asymptotically periodic case. We emphasize that the only difference between the potentials $V_{i, o} (x), λ_{o} (x)$ and $V_{i} (x), λ (x)$ is the periodicity required to $V_{i, o} (x)$ and $λ_{o} (x)$ . Thus, if $V_{i} (x)$ and $λ (x)$ are periodic potentials, we can make use of Theorems 1.1 and 1.2 to get a ground state solution for System ( $S^{μ}$ ). Let us suppose that they are not periodic.

Associated to System ( $S^{μ}$ ), we have the following energy functional $\begin{matrix} I_{μ} (u, v) = \frac{1}{2} ({‖ (u, v) ‖}_{E}^{2} - 2 \int_{R^{N}} λ (x) u v d x) - \frac{μ}{p} ‖ u ‖_{p}^{p} - \frac{1}{q} ‖ v ‖_{q}^{q} . \end{matrix}$

The Nehari manifold associated to System ( $S^{μ}$ ) is defined by $\begin{matrix} N_{μ} = {(u, v) \in E ∖ {(0, 0)} : ⟨ I_{μ}^{'} (u, v), (u, v) ⟩ = 0}, \end{matrix}$ and the Nehari energy level is given by $c_{N_{μ}} = {inf}_{N_{μ}} I_{μ} (u, v)$ . Arguing as before, we deduce that $\begin{matrix} I_{μ} (u, v) ⩾ (\frac{1}{2} - \frac{1}{p}) (1 - δ) {‖ (u, v) ‖}_{E}^{2} ⩾ (\frac{1}{2} - \frac{1}{p}) (1 - δ) α > 0, for all (u, v) \in N_{μ} . \end{matrix}$ Hence, $c_{N_{μ}} > 0$ . The next step is to establish a relation between the energy levels $c_{N_{μ, o}}$ and $c_{N_{μ}}$ .

Lemma 5.1.
$c_{N_{μ}} < c_{N_{μ, o}}$ .
Proof.
Let $(u_{0}, v_{0}) \in N_{μ, o}$ be the nonnegative ground state solution for System ( $S_{o}^{μ}$ ). It is easy to see that Lemma 2.4 works for $I_{μ}$ and $N_{μ}$ . Thus, there exists a unique $t_{μ} > 0$ , depending on μ and $(u_{0}, v_{0})$ , such that $(t_{μ} u_{0}, t_{μ} v_{0}) \in N_{μ}$ . By using ( $V_{4}$ ) we get $\begin{array}{l} \int_{R^{N}} [(V_{1} (x) - V_{1, o} (x)) u_{0}^{2} + (V_{2} (x) - V_{2, o} (x)) v_{0}^{2} + (λ_{o} (x) - λ (x)) u_{0} v_{0}] d x < 0 . \end{array}$ Therefore, $I_{μ} (t_{μ} u_{0}, t_{μ} v_{0}) - I_{μ, o} (t_{μ} u_{0}, t_{μ} v_{0}) < 0$ . Since $(u_{0}, v_{0})$ is a ground state for System ( $S_{o}^{μ}$ ) we can use Lemma 2.4 to deduce that $\begin{matrix} c_{N_{μ}} ⩽ I_{μ} (t_{μ} u_{0}, t_{μ} v_{0}) < I_{μ, o} (t_{μ} u_{0}, t_{μ} v_{0}) ⩽ max_{t ⩾ 0} I_{μ, o} (t u_{0}, t v_{0}) = I_{μ, o} (u_{0}, v_{0}) = c_{N_{μ, o}}, \end{matrix}$ which finishes the proof. □

Let ${(u_{n}, v_{n})}_{n} \subset N_{μ}$ be the minimizing sequence satisfying $\begin{matrix} (16) & I_{μ} (u_{n}, v_{n}) \to c_{N_{μ}} and I_{μ}^{'} (u_{n}, v_{n}) \to 0 . \end{matrix}$ Since ${(u_{n}, v_{n})}_{n}$ is a bounded sequence in E, we may assume up to a subsequence that $(u_{n}, v_{n}) ⇀ (u_{0}, v_{0})$ weakly in E. The main difficulty here is to prove that the weak limit is nontrivial. Proposition 5.2.
The weak limit $(u_{0}, v_{0})$ of the minimizing sequence ${(u_{n}, v_{n})}_{n}$ is nontrivial.
Proof.
We suppose by contradiction that $(u_{0}, v_{0}) = (0, 0)$ . We may assume that
$u_{n} \to 0$ and $v_{n} \to 0$ strongly in $L_{loc}^{p} (R^{N})$ , for all $2 ⩽ p < 2^{}$ ;

$u_{n} (x) \to 0$ and $v_{n} (x) \to 0$ almost everywhere in $R^{N}$ .
It follows from assumption ( $V_{4}$ ) that for any $ε > 0$ there exists $R > 0$ such that $\begin{matrix} (17) & | V_{1, o} (x) - V_{1} (x) | < ε, | V_{2, o} (x) - V_{2} (x) | < ε, | λ (x) - λ_{o} (x) | < ε, for | x | ⩾ R . \end{matrix}$ By using (17) and the local convergence, for any $ε > 0$ , there exists $n_{0} \in N$ such that $\begin{array}{l} | \int_{R^{N}} (V_{1, o} (x) - V_{1} (x)) u_{n}^{2} d x | ⩽ (‖ V_{1} ‖_{L^{\infty} (B_{R} (0))} + ‖ V_{1, o} ‖_{L^{\infty} (B_{R} (0))}) ε + C ε, \\ | \int_{R^{N}} (V_{2, o} (x) - V_{2} (x)) v_{n}^{2} d x | ⩽ (‖ V_{2, o} ‖_{L^{\infty} (B_{R} (0))} + ‖ V_{2} ‖_{L^{\infty} (B_{R} (0))}) ε + C ε, \\ | \int_{R^{N}} (λ (x) - λ_{o} (x)) u_{n} v_{n} d x | ⩽ (‖ λ ‖_{L^{\infty} (B_{R} (0))} + ‖ λ_{o} ‖_{L_{loc}^{\infty} (B_{R} (0))}) ε + C ε, \end{array}$ for all $n ⩾ \tilde{n_{0}}$ . Therefore, we can conclude that $\begin{matrix} I_{μ, o} (u_{n}, v_{n}) - I_{μ} (u_{n}, v_{n}) = o_{n} (1) and ⟨ I_{μ, o}^{'} (u_{n}, v_{n}), (u_{n}, v_{n}) ⟩ - ⟨ I_{μ}^{'} (u_{n}, v_{n}), (u_{n}, v_{n}) ⟩ = o_{n} (1), \end{matrix}$ which jointly with (16) implies that $\begin{matrix} (18) & I_{μ, o} (u_{n}, v_{n}) = c_{N_{μ}} + o_{n} (1) and ⟨ I_{μ, o}^{'} (u_{n}, v_{n}), (u_{n}, v_{n}) ⟩ = o_{n} (1) . \end{matrix}$ By using Lemma 2.4 we obtain a sequence ${(t_{n})}_{n} \subset (0, + \infty)$ such that ${(t_{n} u_{n}, t_{n} v_{n})}_{n} \subset N_{μ, o}$ . Claim 1.
${lim sup}_{n \to + \infty} t_{n} ⩽ 1$ .

Arguing by contradiction, we suppose that there exists $ε_{0} > 0$ such that, up to a subsequence, we have $t_{n} ⩾ 1 + ε_{0}$ , for all $n \in N$ . Thus, using (18) and the fact that $(t_{n} u_{n}, t_{n} v_{n}) \subset N_{μ, o}$ we get $\begin{matrix} (t_{n}^{p - 2} - 1) μ ‖ u_{n} ‖_{p}^{p} + (t_{n}^{q - 2} - 1) ‖ v_{n} ‖_{q}^{q} = o_{n} (1), \end{matrix}$ which together with $t_{n} ⩾ 1 + ε_{0}$ implies that $\begin{matrix} (19) & ({(1 + ε_{0})}^{p - 2} - 1) μ ‖ u_{n} ‖_{p}^{p} + ({(1 + ε_{0})}^{q - 2} - 1) ‖ v_{n} ‖_{q}^{q} ⩽ o_{n} (1) . \end{matrix}$ Similarly to the proof of Theorems 1.1 and 1.2, we define $({\tilde{u}}_{n} (x), {\tilde{v}}_{n} (x)) = (u_{n} (x + y_{n}), v_{n} (x + y_{n}))$ . It follows from assumption ( $V_{4}$ ) that $V_{1}, V_{2} \in L^{\infty} (R^{N})$ . Using the continuous embedding $E_{i} ↪ H^{1} (R^{N})$ we can deduce that ${({\tilde{u}}_{n}, {\tilde{v}}_{n})}_{n}$ is bounded in E. Thus, up to a subsequence, we may consider $({\tilde{u}}_{n}, {\tilde{v}}_{n}) ⇀ (\tilde{u}, \tilde{v})$ weakly in E. Therefore, $\begin{matrix} (20) & lim_{n \to + \infty} \int_{B_{R} (0)} ({\tilde{u}}_{n}^{2} + {\tilde{v}}_{n}^{2}) d x = lim_{n \to + \infty} \int_{B_{R} (y_{n})} (u_{n}^{2} + v_{n}^{2}) d x ⩾ β > 0, \end{matrix}$ which implies $(\tilde{u}, \tilde{v}) \neq (0, 0)$ . We point out that in the critical case, when $q = 2^{}$ , (20) holds for parameters $μ ⩾ μ_{0}$ , where $μ_{0}$ was introduced in Lemma 4.1. Thus, by using (19) and the semicontinuity of the norm, we get $\begin{matrix} 0 < ({(1 + ε_{0})}^{p - 2} - 1) μ ‖ \tilde{u} ‖_{p}^{p} + ({(1 + ε_{0})}^{q - 2} - 1) ‖ \tilde{v} ‖_{q}^{q} ⩽ o_{n} (1), \end{matrix}$ which is not possible and finishes the proof of Claim 1. Claim 2.
There exists $n_{0} \in N$ such that $t_{n} ⩾ 1$ , for $n ⩾ n_{0}$ .

In fact, arguing by contradiction, we suppose that up to a subsequence, $t_{n} < 1$ . Since ${(t_{n} u_{n}, t_{n} v_{n})}_{n} \subset N_{μ, o}$ we have that $\begin{matrix} c_{N_{μ, o}} ⩽ \frac{p - 2}{2 p} μ t_{n}^{p} ‖ u_{n} ‖_{p}^{p} + \frac{q - 2}{2 q} t_{n}^{q} ‖ v ‖_{q}^{q} ⩽ \frac{p - 2}{2 p} μ ‖ u_{n} ‖_{p}^{p} + \frac{q - 2}{2 q} ‖ v ‖_{q}^{q} = c_{N_{μ}} + o_{n} (1) . \end{matrix}$ Therefore, $c_{N_{μ, o}} ⩽ c_{N_{μ}}$ which contradicts Lemma 5.1 and finishes the proof of Claim 2.

Combining Claims 1 and 2 we deduce that $\begin{matrix} I_{μ, o} (t_{n} u_{n}, t_{n} v_{n}) - I_{μ, o} (u_{n}, v_{n}) = o_{n} (1) . \end{matrix}$ Thus, it follows from (18) that $\begin{matrix} c_{N_{μ, o}} ⩽ I_{μ, o} (t_{n} u_{n}, t_{n} v_{n}) = I_{μ, o} (u_{n}, v_{n}) + o_{n} (1) = c_{N_{μ}} + o_{n} (1), \end{matrix}$ which contradicts Lemma 5.1. Therefore, $(u_{0}, v_{0}) \neq (0, 0)$ . □
Proof of Theorem 1.3 completed.
Since $(u_{0}, v_{0})$ is a nontrivial point of the energy functional I, it follows that $(u_{0}, v_{0}) \in N_{μ}$ . Therefore, we have $c_{N_{μ}} ⩽ I_{μ} (u_{0}, v_{0})$ . On the other hand, using the semicontinuity of the norm we deduce that $\begin{array}{rcl} c_{N_{μ}} + o_{n} (1) & = & (\frac{1}{2} - \frac{1}{p}) μ ‖ u_{n} ‖_{p}^{p} + (\frac{1}{2} - \frac{1}{q}) ‖ v_{n} ‖_{q}^{q} \\ ⩾ & (\frac{1}{2} - \frac{1}{p}) μ ‖ u_{0} ‖_{p}^{p} + (\frac{1}{2} - \frac{1}{q}) ‖ v_{0} ‖_{q}^{q} + o_{n} (1) \\ = & I_{μ} (u_{0}, v_{0}) + o_{n} (1) . \end{array}$ Hence, $c_{N_{μ}} ⩾ I_{μ} (u_{0}, v_{0})$ . Therefore $I_{μ} (u_{0}, v_{0}) = c_{N_{μ}}$ . Repeating the same argument used in the proof of Theorem 1.1, we can deduce that there exists $t_{μ} > 0$ such that $(t_{μ} | u_{0} |, t_{μ} | v_{0} |) \in N_{μ}$ is a positive ground state solution for System ( $S^{μ}$ ) which finishes the proof of Theorem 1.3. □

6. Proof of Theorem 1.4

In this section we deal of the following coupled system $\begin{matrix} (21) & \{\begin{matrix} - Δ u + V_{1} (x) u = μ | u |^{2^{*} - 2} u + λ (x) v, & x \in R^{N}, \\ - Δ v + V_{2} (x) v = | v |^{2^{*} - 2} v + λ (x) u, & x \in R^{N} . \end{matrix} \end{matrix}$ In order to obtain a nonexistence result we prove the following Pohozaev identity.

Lemma 6.1.

Suppose $N ⩾ 3$ and let $(u, v) \in E$ be a classical solution of ( 21 ). Then, $(u, v)$ satisfies the following Pohozaev identity: $\begin{array}{l} \int_{R^{N}} (| \nabla u |^{2} + | \nabla v |^{2}) d x \\ = \int_{R^{N}} (μ | u |^{2^{*}} + | v |^{2^{*}} + 2^{*} λ (x) u v) d x + \int_{R^{N}} ⟨ \nabla λ (x), x ⟩ u v d x \\ - \frac{2^{*}}{2} \int_{R^{N}} (V_{1} (x) u^{2} + V_{2} (x) v^{2}) d x - \frac{1}{N - 2} \int_{R^{N}} (⟨ \nabla V_{1} (x), x ⟩ u^{2} + ⟨ \nabla V_{2} (x), x ⟩ v^{2}) d x . \end{array}$

Proof.

In order to get this Pohozaev identity we adapt some ideas from [20, Theorem B.3]. Let $(u, v) \in E$ be a classical solution of the system (21) and let us denote $\begin{matrix} f (x, u, v) = - V_{1} (x) u + μ | u |^{2^{*} - 2} u + λ (x) v and g (x, u, v) = - V_{2} (x) v + | v |^{2^{*} - 2} v + λ (x) u . \end{matrix}$ We consider the cut-off function $ψ \in C_{0}^{\infty} (R)$ defined by $ψ (t) = 1$ if $| t | ⩽ 1$ , $ψ (t) = 0$ if $| t | ⩾ 2$ and $| ψ^{'} (t) | ⩽ C$ , for some $C > 0$ . We define $ψ_{n} (x) = ψ (| x |^{2} / n^{2})$ and we note that $\begin{matrix} \nabla ψ_{n} (x) = \frac{2}{n^{2}} ψ^{'} (\frac{| x |^{2}}{n^{2}}) x . \end{matrix}$ Multiplying the first equation in (21) by the factor $⟨ \nabla u, x ⟩ ψ_{n}$ , the second equation by the factor $⟨ \nabla v, x ⟩ ψ_{n}$ , summing and integrating we get $\begin{matrix} (22) & \int_{R^{N}} (Δ u ⟨ \nabla u, x ⟩ + Δ v ⟨ \nabla v, x ⟩) ψ_{n} d x = \int_{R^{N}} (f (x, u, v) ⟨ \nabla u, x ⟩ + g (x, u, v) ⟨ \nabla v, x ⟩) ψ_{n} d x . \end{matrix}$ The idea is to take the limit as $n \to + \infty$ in (22). In order to calculate the limit in the left-hand side of (22), we note that $\begin{array}{l} ⟨ \nabla u, x ⟩ ψ_{n} Δ u = & div (ψ_{n} H (x, u)) + \frac{N - 2}{2} ψ_{n} | \nabla u |^{2} \\ (23) & + \frac{| \nabla u |^{2}}{2} ⟨ \nabla ψ_{n}, x ⟩ - ⟨ \nabla u, x ⟩ ⟨ \nabla ψ_{n}, \nabla u ⟩, \end{array}$ where $H (x, u) = ⟨ \nabla u, x ⟩ \nabla u - (| \nabla u |^{2} / 2) x$ . Therefore, integrating (23) and using Lebesgue dominated convergence theorem, we conclude that $\begin{matrix} (24) & - lim_{n \to \infty} \int_{R^{N}} ⟨ \nabla u, x ⟩ ψ_{n} Δ u d x = - \frac{N - 2}{2} \int_{R^{N}} | \nabla u |^{2} d x . \end{matrix}$ Analogously, we can deduce the limit $\begin{matrix} (25) & - lim_{n \to \infty} \int_{R^{N}} ⟨ \nabla v, x ⟩ ψ_{n} Δ v d x = - \frac{N - 2}{2} \int_{R^{N}} | \nabla v |^{2} d x . \end{matrix}$ In order to calculate the right-hand side, we note that $\begin{matrix} div (ψ_{n} F (x, u, v) x) = ψ_{n} ⟨ \nabla F (x, u, v), x ⟩ + F (x, u, v) ⟨ \nabla ψ_{n}, x ⟩ + N ψ_{n} F (x, u, v), \end{matrix}$ where $F (x, u, v) = - (1 / 2) V_{1} (x) u^{2} + (μ / 2^{*}) | u |^{2^{*}} + λ (x) u v$ . Hence, we can deduce that $\begin{array}{l} \int_{R^{N}} f (x, u, v) ⟨ \nabla u, x ⟩ ψ_{n} d x \\ = \int_{R^{N}} (div (ψ_{n} F (x, u, v) x) - F (x, u, v) ⟨ \nabla ψ_{n}, x ⟩ ψ_{n}) d x \\ + \int_{R^{N}} (\frac{1}{2} ⟨ \nabla V_{1} (x), u ⟩ u^{2} - N F (x, u, v) ψ_{n} - ⟨ \nabla λ (x), x ⟩ u v - ⟨ λ (x) u \nabla v, x ⟩) ψ_{n} d x . \end{array}$ Analogously, denoting $G (x, u, v) = - \frac{1}{2} V_{2} (x) v^{2} + \frac{1}{2^{*}} | v |^{2^{*}} + λ (x) u v$ , we can deduce that $\begin{array}{l} \int_{R^{N}} g (x, u, v) ⟨ \nabla v, x ⟩ ψ_{n} d x \\ = \int_{R^{N}} (div (ψ_{n} G (x, u, v) x) - G (x, u, v) ⟨ \nabla ψ_{n}, x ⟩ ψ_{n}) d x \\ + \int_{R^{N}} (\frac{1}{2} ⟨ \nabla V_{2} (x), v ⟩ v^{2} - N G (x, u, v) ψ_{n} - ⟨ \nabla λ (x), x ⟩ u v - ⟨ λ (x) v \nabla u, x ⟩) ψ_{n} d x . \end{array}$ By using integration by parts we have that $\begin{array}{l} - \int_{R^{N}} λ (x) ⟨ u \nabla v + v \nabla u, x ⟩ ψ_{n} d x \\ = \int_{B_{2 n} (0)} (⟨ \nabla ψ_{n}, x ⟩ λ (x) u v + ⟨ \nabla λ (x), x ⟩ ψ_{n} u v + N ψ_{n} λ (x) u v) d x, \end{array}$ which implies that $\begin{matrix} lim_{n \to \infty} \int_{R^{N}} λ (x) ⟨ u \nabla v + v \nabla u, x ⟩ ψ_{n} d x = - \int_{R^{N}} ⟨ \nabla λ (x), x ⟩ u v d x - N \int_{R^{N}} λ (x) u v d x . \end{matrix}$ Therefore, using the Lebesgue dominated convergence theorem in the same way as we used when we calculate the left-hand side, we obtain $\begin{array}{l} lim_{n \to \infty} \int_{R^{N}} (f (x, u, v) ⟨ \nabla u, x ⟩ + g (x, u, v) ⟨ \nabla v, x ⟩) ψ_{n} d x \\ = - N \int_{R^{N}} (F (x, u, v) + G (x, u, v)) d x \\ + \frac{1}{2} \int_{R^{N}} (⟨ \nabla V_{1} (x), x ⟩ u^{2} + ⟨ \nabla V_{2} (x), x ⟩ v^{2}) d x \\ - \int_{R^{N}} ⟨ \nabla λ (x), x ⟩ u v d x + N \int_{R^{N}} λ (x) u v d x . \end{array}$ Replacing $F (x, u, v)$ and $G (x, u, v)$ in the equation above, we get the right-hand side of (22) which finishes the proof. □

Proof of Theorem 1.4 completed.

Let $(u, v) \in E$ be a positive classical solution of (21). By the definition of weak solution we obtain $\begin{matrix} (26) & \int_{R^{N}} (| \nabla u |^{2} + V_{1} (x) u^{2} + | \nabla v |^{2} + V_{2} (x) v^{2}) d x = \int_{R^{N}} (μ | u |^{2^{*}} + | v |^{2^{*}} + 2 λ (x) u v) d x . \end{matrix}$ Combining (26) with the Pohozaev identity obtained in Lemma 6.1, we have $\begin{array}{l} 0 = & (1 - \frac{2^{*}}{2}) \int_{R^{N}} (V_{1} (x) u^{2} + V_{2} (x) v^{2} - 2 λ (x) u v) d x + \int_{R^{N}} ⟨ \nabla λ (x), x ⟩ u v d x \\ (27) & - \frac{1}{N - 2} \int_{R^{N}} (⟨ \nabla V_{1} (x), x ⟩ u^{2} + ⟨ \nabla V_{2} (x), x ⟩ v^{2}) d x . \end{array}$ Multiplying (27) by the factor $- (N - 2) / 2$ , we get $\begin{array}{l} \int_{R^{N}} (V_{1} (x) u^{2} + V_{2} (x) v^{2} - 2 λ (x) u v) d x = & \frac{N - 2}{2} \int_{R^{N}} ⟨ \nabla λ (x), x ⟩ u v d x \\ - \frac{1}{2} \int_{R^{N}} (⟨ \nabla V_{1} (x), x ⟩ u^{2} + ⟨ \nabla V_{2} (x), x ⟩ v^{2}) d x . \end{array}$ Thus, it follows from assumptions ( $V_{7}$ ) and ( $V_{8}$ ) that $\begin{matrix} \int_{R^{N}} (V_{1} (x) u^{2} + V_{2} (x) v^{2} - 2 λ (x) u v) d x ⩽ 0 . \end{matrix}$ On the other hand, by assumption ( $V_{3}$ ) we get $\begin{matrix} \int_{R^{N}} (V_{1} (x) u^{2} + V_{2} (x) v^{2} - 2 λ (x) u v) d x ⩾ 0 . \end{matrix}$ Thus, we conclude that $\begin{matrix} \int_{R^{N}} (V_{1} (x) u^{2} + V_{2} (x) v^{2} - 2 λ (x) u v) d x = 0 . \end{matrix}$ Therefore, we finally deduce that $\begin{array}{rcl} 0 & ⩽ & \int_{R^{N}} (V_{1} (x) u^{2} - 2 \sqrt{V_{1} (x) V_{2} (x)} u v + V_{2} (x) v^{2}) d x \\ ⩽ & \int_{R^{N}} (V_{1} (x) u^{2} + V_{2} (x) v^{2} - \frac{2}{δ} λ (x) u v) d x \\ < & \int_{R^{N}} (V_{1} (x) u^{2} + V_{2} (x) v^{2} - 2 λ (x) u v) d x = 0, \end{array}$ which is a contradiction and this finishes the proof of Theorem 1.4. □

Footnotes

Acknowledgements

The authors would like to express their sincere gratitude to the referee for carefully reading the manuscript and valuable comments and suggestions.

References

Akhmediev and

Ankiewicz, Novel soliton states and bifurcation phenomena in nonlinear fiber couplers, Phys. Rev. Lett. 70 (1993), 2395–2398. doi:10.1103/PhysRevLett.70.2395.

Ambrosetti,

Cerami and

Ruiz, Solitons of linearly coupled systems of semilinear non-autonomous equations on

R^{N}

, J. Funct. Anal. 254 (2008), 2816–2845. doi:10.1016/j.jfa.2007.11.013.

Bahri and

Y.Y.

Li, On a min-max procedure for the existence of a positive solution for certain scalar field equations in

R^{N}

, Rev. Mat. Iberoam. 6 (1990), 1–16. doi:10.4171/RMI/92.

Bahri and

Lions, On the existence of a positive solution of semilinear elliptic equations in unbounded domains, Ann. Inst. H. Poincaré Anal. Non Linéaire 14 (1997), 365–413. doi:10.1016/S0294-1449(97)80142-4.

Bartsch and

Z.Q.

Wang, Existence and multiplicity results for some superlinear elliptic problems on

R^{N}

, Comm. Part. Diff. Eq. 20 (1995), 1725–1741. doi:10.1080/03605309508821149.

Byeon and

Z.Q.

Wang, Standing waves with a critical frequency for nonlinear Schrödinger equations, Arch. Ration. Mech. Anal. 165 (2002), 295–316. doi:10.1007/s00205-002-0225-6.

Chen and

Zou, On coupled systems of Schrödinger equations, Adv. Differential Equations 16 (2011), 755–800.

Chen and

Zou, Ground states for a system of Schrödinger equations with critical exponent, J. Funct. Anal. 262 (2012), 3091–3107. doi:10.1016/j.jfa.2012.01.001.

Ekeland, On the variational principle, J. Math. Anal. Appl. 47 (1974), 324–353. doi:10.1016/0022-247X(74)90025-0.

10.

Furtado,

L.A.

Maia and

E.S.

Medeiros, Positive and nodal solutions for a nonlinear Schrödinger equation with indefinite potential, Adv. Nonlinear Stud. 8 (2008), 353–373. doi:10.1515/ans-2008-0207.

11.

Gilbarg and

Trudinger, Elliptic Partial Differential Equations of Second Order, Springer-Verlag, New York, 1983.

12.

Guo and

Zou, On a class of coupled Schrödinger systems with critical Sobolev exponent growth, Math. Methods Appl. Sci. 39 (2016), 1730–1746. doi:10.1002/mma.3598.

13.

Jost, Partial Differential Equations, Springer-Verlag, New York, 2002.

14.

Li and

X.H.

Tang, Nehari-type ground state solutions for Schrödinger equations including critical exponent, Appl. Math. Lett. 37 (2014), 101–106. doi:10.1016/j.aml.2014.06.003.

15.

P.L.

Lions, The concentration-compactness principle in the calculus of variations. The locally compact case, Ann. Inst. H. Poincaré Anal. Non Linéaire 1 (1984), 109–145, 223–283. doi:10.1016/S0294-1449(16)30422-X.

16.

L.A.

Maia,

Montefusco and

Pellacci, Positive solutions for a weakly coupled nonlinear Schrödinger system, J. Differential Equations 229 (2006), 743–767. doi:10.1016/j.jde.2006.07.002.

17.

Pomponio, Coupled nonlinear Schrödinger systems with potentials, J. Differential Equation 227 (2006), 258–281. doi:10.1016/j.jde.2005.09.002.

18.

Rabinowitz, On a class of nonlinear Schrödinger equations, Z. Angew. Math. Phys. 43 (1992), 270–291. doi:10.1007/BF00946631.

19.

Z.Q.

Wang and

Xia, Ground states for nonlinear Schrödinger equations with a sign-changing potential well, Adv. Nonlinear Stud. 15 (2015), 749–762. doi:10.1515/ans-2015-0401.

20.

Willem, Minimax Theorems, Birkhäuser, Boston, 1996.

21.

Zhang,

Xu and

Zhang, Existence of positive ground states for some nonlinear Schrödinger systems Bound. Value Probl. (2013), 16 pp.

22.

Zhang, Stability of standing waves for nonlinear Schrödinger equations with unbounded potentials, Z. Angew. Math. Phys. (2000), 498–503. doi:10.1007/PL00001512.