Index Terms- Shannon theory, channel output statistics, re- solvability, random number ... achievability part of the resolvability thereom (which gives an upper bound to ... Definition 1: A channel W with input and output alpha- bets, A and B, ...
152
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 39, NO. 3, MAY 1993
Approximation Theory of Output Statistics Te Sun Han, Fellow, IEEE, and Sergio VerdG, Fellow, IEEE
Abstract-Given a channel and an input process, the minimum randomness of those input processes whose output statistics approximate the original output statistics with arbitrary accuracy is studied. The notion of resolvability of a channel, defined as the number of random bits required per channel use in order to generate an input that achieves arbitrarily accurate approximation of the output statistics for any given input process, is introduced. A general formula for resolvability that holds regardless of the channel memory structure, is obtained. It is shown that, for most channels, resolvability is equal to Shannon capacity. By-products of the analysis are a general formula for the minimum achievable (fixed-length) source coding rate of any finite-alphabet source, and a strong converse of the identification coding theorem, which holds for any channel that satisfies the strong converse of the channel coding theorem.
Index Terms- Shannon theory, channel output statistics, resolvability, random number generation complexity; channel capacity, noiseless source coding theorem, identification via channels. I.
INTRODUCTION
T
0 MOTIVATE the problem studied in this paper let us consider the computer simulation of stochastic systems. Usually, the objective of the simulation is to compute a set of statistics of the response of the system to a given “realworld” input random process. To accomplish this, a sample path of the input random process is generated and empirical estimates of the desired output statistics are computed from the output sample path. A random number generator is used to generate the input sample path and an important question is how many random bits are required per input sample. The answer would depend only on the given “real-world” input statistics if the objective were to reproduce those statistics exactly (in which case an infinite number of bits per sample would be required if the input distribution is continuous, for example). However, the real objective is to approximate the output statistics. Therefore, the required number of random bits depends not only on the input statistics but on the degree of approximation required for the output statistics, and on the system itself. In this paper, we are interested in the approximation of output statistics (via an alternative input process) with arbitrary accuracy, in the sense that the distance between the finite-dimensional statistics of the true output process and the approximated output process is required to vanish asymptotically. This leads to the introduction of a new Manuscript received February 7, 1992; revised September 18, 1992. This work was supported in part by the U.S. Office of Naval Research under Grant N00014-90-J-1734 and in part by the NEC Corp. under its grant program. T. S. Han is with the Graduate School of Information Systems, University of Electra-Communications, Tokyo 182, Japan. S. Verdti is with the Department of Electrical Engineering, Princeton University, Princeton, NJ 08544. IEEE Log Number 9206960. 0018-9448/93$03.00
concept in the Shannon theory; the resolvability of a system (channel) defined as the number of random bits per input sample required to achieve arbitrarily accurate approximation of the output statistics regardless of the actual input process. Intuitively, we can anticipate that the resolvability of a system will depend on how “noisy” it is. A coarse approximation of the input statistics whose generation requires comparatively few bits will be good enough when the system is very noisy, because, then, the output cannot reflect any fine detail contained in the input distribution. Although the problem of approximation of output statistics involves no codes of any sort or the transmission/reproduction of information, its analysis and results turn out to be Shannon theoretic in nature. In fact, our main conclusion is that (for most systems) resolvability is equal to Shannon capacity. In order to make the notion of resolvability precise, we need to specify the “distance” measure between true and approximated output statistics and the “complexity” measure of random number generation. Our main, but not exclusive, focus is on the II-distance (or variational distance) and on the worst-case measure of randomness, respectively. This complexity measure of a random variable is equal to the number of random bits required to generate every possible realization of the random variable; we refer to it as the resolution of the random variable and we show how to obtain it from the probability distribution. The alternative, average randomness measure is known to equal the entropy plus at most two bits [ll], and it leads to the associated notion of mean-resolvability. Section II introduces the main definitions. The class of channels we consider is very general. To keep the development as simple as possible we restrict attention to channels with finite input/output alphabets. However, most of the proofs do not rely on that assumption, and it is specifically pointed out when this is not the case. In addition to allowing channels with arbitrary memory structure, we deal with completely general input processes, in particular, neither ergodicity nor stationarity assumptions are imposed. Further motivation for the notions of resolvability and mean-resolvability is given in Section III. Section IV gives a general formula for the resolvability of a channel. The achievability part of the resolvability thereom (which gives an upper bound to resolvability) holds for any channel, regardless of its memory structure or the finiteness of the input/output alphabets. The finiteness of the input set is the only substantive restriction under which the converse part (which lower bounds resolvability) is shown in Section IV via Lemma 6. The approximation of output statistics has intrinisic connections with the following three major problems in the Shannon theory: (noiseless) source coding, channel coding and identi0 1993 IEEE
753
HAN AND VERD& APPROXIMATION THEORY OF OUTPUT STATISTICS
fication via channels [l]. As a by-product of our resolvability results, we find in Section III a very general formula for the minimum achievable fixed-length source coding rate that holds for any finite-alphabet source thereby dispensing with the classical assumptions of ergodicity and stationarity. In Section V, we show that as long as the channel satisfies the strong converse to the channel coding theorem, the resolvability formula found in Section IV is equal to the Shannon capacity. As a simple consequence of the achievability part of the resolvability theorem, we show in Section VI a general strong converse to the identification coding theorem, which was known to hold only for discrete memoryless channels [7]. This result implies that the identification capacity is guaranteed to equal the Shannon capacity for any finite-input channel that satisfies the strong converse to the Shannon channel coding theorem. The more appropriate kind (average or worst-case) of complexity measure will depend on the specific application. For example, in single sample-path simulations, the worst-case measure may be preferable. At any rate, the limited study in Section VII indicates that in every case we consider, the mean-resolvability is also equal to the Shannon capacity of the system. Similarly, the results presented in Section VIII evidence that the main conclusions on resolvability (established in previous sections) also hold when the variational-distance approximation criterion is replaced by the normalized divergence. Section VIII concludes with the proof of a folk theorem which fits naturally within the approximation theory of output statistics: the output distribution due to any good channel code must approximate the output distribution due to the capacityachieving input. Although the problem treated in this paper is new, it is interesting to note two previous information-theoretic contributions related to the notion of quantifying the minimum complexity of a randomness source required to approximate some given distribution. In one of the approaches to measure the common randomness between two dependent random variables proposed in [21], the randomness source is the input to two independent memoryless random transformations, the outputs of which are required to have a joint distribution which approximates (in normalized divergence) the nth product of the given joint distribution, The class of channels whose transition probabilities can be approximated (in &distance) by slidingblock transformations of the input and an independent noise source are studied in [13], and the minimum entropy rate of the independent noise source required for accurate approximation is shown to be the maximum conditional output entropy over all stationary inputs.
In order to describe the statistics of input/output processes, we will use the sequence of finite-dimensional distributions’ {Xn = (X,‘“‘, . .. ,X?))}r=,, which is abbreviated as X. The following notation will be used for the output distribution when the input is distributed according to Q”: Q”W”(y”)
=
c
W”(y”
1 ?)&“(a?)
0 ilm
P[A,
> /3 + E] = 0.
Analogously, the liminf in probability is the largest extended real number Q! such that for all e > 0, limn+cr, P[A, 5 Cl-E] = 0. Note that a sequence of random variables converges in probability to a constant, if and only if its limsup in probability is equal to its liminf in probability. The limsup in probability [resp. liminf in probability] of the sequence of random variables {(l/n)ix%w(Xn, Yn)}rZ1 will be referred to as the sup-information rate [resp. infinformation rate] of the pair (X, Y) and will be denoted as 7(X; Y) [resp. 1(X; Y)]. The mutual information rate of (X, Y), if it exists, is the limit 1(X; Y) = lim I1(X”; 71’00 n
Y”).
Although convergence in probability does not necessarily imply convergence of the means, (e.g., [15, p. 135]), in most cases of information-theoretic interest that implication does indeed hold in the context of information rates. Lemma 1: For any channel with finite input alphabet, if I(X; Y) = L(X; Y), (i.e., the information spectrum converges in probability to a constant), then 7(X; Y) = 1(X; Y) = l(X;
Y),
II. PRELIMINARIES This section introduces the basic notation and fundamental concepts as well as several properties to be used in the sequel. Definition 1: A channel W with input and output alphabets, A and B, respectively, is a sequence of conditional distributions w = {w”(yn
1 2”) = Pyn,X”(yn
12”); (cz?, y”) E A” x B”},“=,.
and the input-output pair (X, Y) is called information stable. Proof: See the Appendix for a proof that hinges on the 0 finiteness of the input alphabet. 1 No consistency restrictions between the channel conditional probabilities, or between the finite-dimensional distributions of the input/output processes are imposed. Thus, “processes” refer to sequences of finite-dimensional distributions, rather than distributions on spaces of infinite dimensional sequences.
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 39, NO. 3, MAY 1993
754
Definition 4 [7]: For any positive integer M,2 a probability distribution P is said to be M - type if
and d(Y”,
P(u) E
0, $-, ;,
. . . ,l
C
I
for all w E R.
)
Py- = Px-kV, P+
sups, E>O
with equality, if and only if P is equiprobable. The information spectrum is upper bounded almost surely by the input (or output) resolution: Lemma 3:
Proof: For every (a?, y”) Px-(x”) > 0,3 we have iX”W”(xn,
= 1.
E A” x B”
y”) 5 log
1 px- (x”)
exp (-R(Xn)),
~I%4 WE0 = 2~5
Q(w)1 P’(E) - Q(WI.
(2.2)
-
(2.3)
Definition 7: Let c 2 0. R is an c-achievable resolution rate for channel W if for every input process X and for all y > 0, there.exists h whose resolution satisfies ;R(Xn)
< R + y
s = sup S(X). X
(2.1)
Definition 6 (e.g., [3]): The variational distance or Iidistance between two distributions P and Q defined on the same measurable space (R, 9) is
Q)=
(2.6)
The main focus of this paper is on the resolvability of systems as defined in Definition 7. In addition, we shall investigate another kind of resolvability results by considering a different randomness measure. Specifically, if in Definition 7, (2.4) is replaced by
where m(z”) is an integer greater than or equal to 1. Thus, 0 the result follows uniting (2.1) and (2.2).
d(P>
= s.
The definitions of achievable resolution rates can be modified so that the defining property applies to a particular input X instead of every input process. In such case, we refer to the corresponding quantities as (c-) achievable resolution rate for X and (E-) resolvability for X, for which we use the notation S,(X) and S(X). It follows from Definition 7 that
such that
and Px-(CC”) = m(C)
= P.&%W”.
If R is an c-achievable resolution rate, for every E > 0, then, we say that R is an achievable resolution rate. By definition, the set of (e-) achievable resolution rates is either empty or a closed interval. The minimum e-achievable resolution rate (resp., achievable resolution rate) is called the c-resolvability (resp., resolvability) of the channel, and it is denoted by S, (resp., S). Note that S, is monotonically nonincreasing in E and
H(P)< z(P)5 R(P)
Y”) < R(X”)]
(2.5)
for all sufficiently large 12, where Y and Y are the output statistics due to input processes X and X, respectively, i.e.,
The number of different M-type distributions on R is upper bounded by IRIM. Definition 5: The resolution R(P) of a probability distribution P is the minimum log M such that P is M-type. (If P is not M-type for any integer M, then R(P) = +oo.) Resolution is a new measure of randomness which is related to conventional measures via the following immediate inequality. Lemma 2: Let H(P) denote the entropy of P and let Z(P) denote the RCnyi entropy of order 0, i.e., the logarithm of the number of points with positive P-mass. Then.
p[iXnWn(Xn,
9’“) < E,
$2”)
(2.7)
then achievable resolution rates become achievable entropy rates and resolvability becomes mean-resolvability. It follows from Lemma 2 that for all E > 0 and X
ax) 5 Se(X)
(2.8)
where 3, and ?? denote (E-) mean-resolvability in parallel with the above definitions of S, and S. It is obvious that 3 = supx 3(X). The motivation for the definitions of resolvability and meanresolvability is further developed in the following section. III. RESOLUTION,
(2.4)
‘The alternative terminology type with denominator M can be found in [2, ch. 121. 3Following common usage in information theory, when the distributions in Definitions 4-6 denote those of random variables, they will b_ereplaced by the random variables themselves, e.g., R(X), H(X), d(Yn, Y”).
< R + y
RANDOM
NUMBER
GENERATION
AND SOURCE CODING
The purpose of this section is to motivate the definitions of resolvability and mean-resolvability introduced in Section II through their relationship with random number generation and noiseless source coding. Along the way, we will show that our resolvability theorems lead to new general results in source coding.
HAN AND VERDti: APPROXIMATION THEORY OF OUTPUT STATISTICS
A. Resolution and Random Number Generation A prime way to quantify the “randomness” of a random variable is via the complexity of its generation with a computer that has access to a basic random experiment which generates equally likely random values, such as fair coin flips, dice, etc. By complexity, we mean the number of random bits that the most efficient algorithm requires in order to generate the random variable. Depending on the algorithm, the required number of random bits may be random itself. For example, consider the generation of the random variable with probability masses P[X = -11 = l/4, P[X = 0] = l/2, P[X = 11 = l/4, with an algorithm such that if the outcome of a fair coin flip is Heads, then the output is 0, and if the outcome is Tails, another fair coin flip is requested in order to decide $1 or -1. On the average this algorithm requires 1.5 coin flips, and in the worst-case 2 coin flips are necessary. Therefore, the complexity measure can take two fundamental forms: worstcase or average (over the range of outcomes of the random variable). First, let us consider the worst-case complexity. A conceptual model for the generation of arbitrary random variables is a deterministic transformation of a random variable uniformly distributed on [0, 11. Although such a random variable cannot be generated by a discrete machine, this model suggests an algorithm for the generation of finitely-valued random variables in a finite-precision computer: a deterministic transformation of the outcome of a random number generator which outputs M equally likely values, in lieu of the uniformly distributed random variable. The lowest value of log M required to generate the random variable (among all possible deterministic transformations) is its worst-case complexity. Other algorithms may require fewer random bits on the average, but not for every possible outcome. It is now easy to recognize that the worst-case complexity of a random variable is equal to its resolution. This is because processing the output of the M-valued random number generator with a deterministic transformation (which is conceptually nothing more than a table lookup) results in a discrete random variable whose probability masses are multiples of l/M, i.e., an M-type. At first sight, it may seem that the use of resolution (as opposed to entropy) in the definition of resolvability is overly stringent. However, this is not the case because that definition is concerned with asymptotic approximation. Analogously, in practice, M may be constrained to be a power of 2; however, this possible modification has no effect on the definition of achievable resolution rates (Definition 7) because it is only concerned with the asymptotic behavior of the ratio of resolution to number of dimensions of the approximating distribution. The average complexity of random variable generation has been studied in the work of Knuth and Yao [ll], which shows that the minimum expected number of fair bits required to generate a random variable lies between its entropy and its entropy plus two bits, (cf. [2, theorem 5.12.31). That lower bound holds even if the basic equally likely random number generator is allowed to be nonbinary. This result is the reason for the choice of entropy as the average complexity measure
755
in the definition of mean-resolvability. Note that the two-bit uncertainty of the Knuth-Yao theorem is inconsequential for the purposes of our (asymptotic) definition.
B. Resolution and Source Coding Having justified the new concepts of resolvability and meanresolvability on the basis of their significance in the complexity of random variable generation, let us now explore their relationship with well-established concepts in the Shannon theory. To this end, in the remainder of this section we will focus on the special case of an identity channel (A = B; W” (y” ( xn) = 1 if xn = y/“), in which our approximation theory becomes one of approximation of source statistics. Suppose we would ike to generate random sequences according to the finite dimensional distributions of some given process X. As we have argued, the worst-case and average number of bits per dimension required are (l/n)R(Xn) and (l/n)H(Xn), respectively. If, however, we are content with reproducing the source statistics within an arbitrarily small tolerance, fewer bits may be needed, asymptotically in the worst case. For example, consider the case of independent flips of a biased coin with tails probability equal to l/~. It is evident that R(Xn) = 00 for every n. However, the asymptotic equipartition property (AEP) states that for any E > 0 and large n, the exp (&(1/x) + nc) typical sequences exhaust most of the probability. If we let M = exp (nh(l/rr) + 2nc) then we can quantize the probability of each of those sequences to a multiple of l/M, thereby achieving a quantization error in each mass of at most l/M. Consequently, the sum of the absolute errors on the typical sequences is exponentially small, and the masses of the atypical sequences can be approximated by zero because of the AEP, thereby yielding an arbitrarily small variational distance between the true and approximating statistics. The resolution rate of the approximating statistics is h(l/7r)+2~. Indeed, in this case S(X) = s(X) = h(l/r), and this reasoning can be applied to any stationary ergodic source to show that S(X) is equal to the entropy rate of X (always in the context of an identity channel). The key to the above procedure to approximate the statistics of the source with finite resolution is the use of repetition. Had we insisted on a uniform approximation to the original statistics we would not have succeeded in bringing the variational distance to negligible levels, because of the small but exponentially significant variation in the probability masses of the typical sequences. By allowing an approximation with a uniform distribution on a collection of M elements with repetition, i.e., an M-type, with large enough M, it is possible to closely track those variations in the probability masses. A nice bonus is that for this approximation procedure to work it is not necessary that the masses of the typical sequences be similar, as dictated by the AEP. This is why the connection between resolvability and source coding is deeper than that provided by the AEP, and transcends stationary ergodic sources. To show this, let us first record the standard definitions of the fundamental limits in fixed-length and variable-length source coding. Definition 8: R is an &-achievable source coding rate for X if for all y > 0 and all sufficiently large n, there exists a
IEEE TRANSACTIONSON INFORMATION THEORY, VOL. 39, NO. 3, MAY 1993
756
P[Xn $z {XT,. . . ,x&)1 L E.
collection of M n-tuples {x’;“, . . . , x%} such that
Choose M’ such that
~10gM~Rfy
exp (nR + 2ny) I M’ I exp (nR + 3ny)
and
R is an achievable (fixed-length) source coding rate for X if it is e-achievable for all 0 < E < 1. T(X) denotes the minimum achievable source coding rate for X. Definition 9: Fix an integer r > 2. R is an achievable variable-length source coding rate for X if for all y > 0 and all sufficiently large n, there exists an r-ary prefix code for X” such that the average codeword length L, satisfies ;Ln
log r < Rfy.
The minimum achievable variable-length source coding rate for X is denoted by T(X). As shown below, in the special case of the identity channel, resolvability and mean-resolvability reduce to the minimum achievable fixed-length and variable-length source coding rates, respectively, for any source. Although quite different from the familiar setting of combined source and channel coding (e.g., no decoding is present at the channel output), the approximation theory of output statistics could be subtitled “source coding via channels” because of the following two results. Theorem 1: For any X and the identity channel,
,
and an arbitrary element xg @ D. ’ We are going to construct an approximation Xn to X” which satisfies the following conditions: a) b) c) d)
Xn is an M/-type, P&6$) 5 t, ]P*n(xr) - Px-(xa)] 5 l/M’, P+(D) + P+(x$) = 1.
It will then follow immediately that R is a 3e-achievable resolution rate, as $2”)
5 R + 37
and M
4xn>
0
5 ClP+(xy)
- Px-(x;)l+
P+(x;;)
i=l +
c
px-(xn)
X%EDC
0 and all sufficiently large n, select D = {x;, +. +, XL} such that 1 log M < R + y, ,n
with 1 5 L 5 M. Since it may be assumed, without loss of generality, that Pp(xT) > 0 for all i = 1,. .. , M we may set IcrJ= 0 ki = [M’Px~~(~r)j - 1 > 0, ki = [M’Pxn (x%)1,
i=
l,*..L,
i = L+
which again guarantees that a)-d) are satisfied.
l,...M, 0
151
HAN AND VERDtl: APPROXIMATION THEORY OF OUTPUT STATISTICS
Theorem 2: For any X and the identity channel, S(X)
= T(X)
= lim SUP~+~ $X”).
Proof: 1) s(X) 5 T(X). Suppose that R is an achievable variable-length source coding rate. Then, Definition 9 states that there exists for all y > 0 and all sufficiently large n a prefix code whose average length L, satisfies
Theorems 1 and 2 show a pleasing parallelism between the resolvability of a process (with the identity channel) and its minimum achievable source coding rate. Theorem 1 and the Shannon-MacMillan theorem lead to the solution of S(X) as the entropy rate of X in the special case of stationary ergodic X. Interestingly, the results of this paper allow us to find the resolvability of any process with the identity channel, and thus a completely general formula for the minimum achievable source-coding rate for any source. Theorem 3: For any X and the identity channel,
;Ln
log T < R + y.
(3.1)
Moreover, the fundamental source coding lower bound for an r-ary code (e.g., [2, theorem 5.3.11) is II
5 L, log T.
< R + y,
(3.3)
concluding that R is an achievable mean-resolution rate for X. 2) T(X) < s(X). Let R be an achievable mean-resolution rate for X. For arbitrary y > 0, 0 < E < l/2 and all sufficiently large n, choose Xn such that (3.3) is satisfied and d(Xn, 2”) < E. On the other hand, there exists an r-ary prefix code for X” with average length bounded by (e.g., [2, theorem 5.4.1) L, log T < H(Xn)
+ log T.
1 Proof: 1) H(X) 5 S(X). w e will argue by contradiction: choose an achievable resolution rate R for X such that for some s>o R+S 0
and 1 _ P+W)
Pxn(2”)
Pxn(Dl log IAl” E
< E
< $/2
I-
1
and consider
+ ; log T 5 $?(a”) +‘t n
z R+S
Select 0 < E < a2, and Xn for all sufficiently large n to satisfy
(3.5)
Using Lemma 4 and (3.3), we obtain
(3.6)
1 1 log PX”(““) n
d(Xn, P)
for all sufficiently large n, thereby proving that R is an achievable variable length source coding rate for X. To that end, all that is required is the continuity of entropy in variational distance: Lemma 4 (3, p. 331: If P and Q are distributions defined on R such that d(P, Q) 5 6’< l/2, then -
2 Q
infinitely often with DO defined as the set of least likely source words:
AL% log r 2 R + 27 n
V(P)
there exists a! > 0 such that
px-(Do)
(3.4)
We want to show that if the above E is chosen sufficiently small then the code satisfies
= H(X),
where H(X) is the sup-entropy rate defined as 7(X; Y) for the identity channel (cf. Definition 3), i.e., the smallest real number ,0 such that for all E > 0
(3.2)
Now, let X = X. Then, d(Xn, Xn) = 0, and (3.1)-(3.2) imply that h(%“) n
S(X) = T(X)
0
n Do) 2 Px-(Do) - Px-(DE) > a - Eli2 > 0,
which holds infinitely often because of (3.6) and ~~‘~px-(D:)
I
c X-CD;
2 E.
Px=(x:“)
P+ (2”) 1 - pxn(xn)
(3.7)
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 39, NO. 3, MAY 1993
758
For those n such that (3.7) holds, we can find x2; E D1 n Do whose P*,-mass satisfies the following lower and upper bounds: P+ (x;) 2 (1 - 2’2)Pxn
This result will be shown by means of an achievability (or direct) theorem which provides an upper bound to resolvability along with a converse theorem which gives a lower bound to resolvability.
(xgn) > 0 A. Direct Resolvability Theorem
and > n1 log Pp(xg) 1
; log P.+(x$) 1
1 + n1 log 1 + El/2
Theorem 4: Every channel W and input process X satisfy S,(X)
L qx;
y>
for any E > 0 where Y is the output of W due to X. if n is sufficiently large. Therefore we have found (an infinite number of) n such that Pj+
1
1 log n P+ p)
L P*n
+ log
> ;R(2”) 1
P+&(X”)
>R+; -
1 1
contradicting Lemma 3. is a special case (identity channel) of the 2) S(X) I mq general direct resolvability result (Theorem 4 in Section q IV). Remark 1: We may consider a modified version of Definition 9 as follows. Let us say that an r-ary variable-length code for X” is an c-prefix code for X” (0 < E < 1) {@(xn))rn~An if there exists a subset D of A” such that Pp (D) 2 1 - 6 is a prefix code. It is easy to check that and {$(x~)),~cD Theorem 2 continues to hold if “all y > 0” and “r-ary prefix code” are replaced by “all y > 0 and 0 < E < 1” and “r-ary c-prefix code” respectively in Definition 9. A general formula for the minimum achievable rate for noiseless source coding without stationarity and ergodicity assumptions has been a longstanding goal. It had been achieved [lo] in the setting of variable-length coding (see Theorem 2). In fixed-length coding, progress towards that goal had been achieved mainly in the context of stationary sources (via the ergodic decomposition theorem, e.g., [6]). A general result that holds for nonstationary/nonergodic sources is stated in4 [9] without introducing the notions of T(X) and H(X). The results established in this section from the standpoint of distribution approximation attain general formulas for both fixed-length and variable-length source coding without recourse to stationarity or ergodicity assumptions. It should be noted that an independent proof of T(X) = g(X) can be obtained by generalizing the proof of the source coding theorem in [3, Theorem 1.11. IV.
Proof: Fix an arbitrary y > 0. According to Definition 7, we have to show the existence of a process x such that lim d(Yn, 9’“) = 0 n-ice and gn is an M-type distribution with M = exp (nT(X; Y) + ny), and Y”, Y” are the output distribution due to X” and zn, respectively. We will construct the approximating input statistics by the Shannon random selection approach. For any collection of (not necessarily distinct) M elements of A”, the distribution constructed by placing l/M mass on each of the members of the collection is an M-type distribution. If each member of the collection is generated randomly and independently with distribution Xn, we will show that the variational distance between Y” and the approximated output averaged over the selection of the M-collection vanishes, and hence there must exist a sequence of realizations for which the variational distance also vanishes. For any { cj E A”, j = 1, +. . , M} denote the output distribution ](Yn) pPn[C1,..‘ ,CM=
~~~n(Yn
I Cj).
(4.1)
3=1
The objective is to show that lim Ed(Y”, n--too
?“[X;,.+.,X$])
= 0,
where the expectation is with respect to i.i.d. (X;, . e. , X&) with common distribution Xn. Instead of using the standard Csiszar-Kullback-Pinsker bound in terms of divergence [3], in order to upper bound d(Y”, Y”[X;, . e. , X$]) we will use the following new bound in terms of the distribution of the log likelihood ratio. Lemma 5: For every p > 0,
RESOLVAL~ILITY THEOREMS
A general formula for the resolvability of any channel in terms of its statistical description is obtained in this section. 4[9] refers to a Nankai University thesis by T. S. Yang for a proof.
d(P, Q) L -?- cL+2p .log e
P(X) 1% Q(x)
where X is distributed according to P.
>P
1 >
HAN AND VERDfJ: APPROXIMATION THEORY OF OUTPUT STATISTICS
759
Proof of Lemma 5: We can write the variational distance as
g}
d(P,Q) = cl 0 5 log XEcl 1
P=(x)- Q(x)1
log $j + Cl XEcl { = 2dl + 2d2,
5 O}
[Q(x)-
where r = (exp p - 1)/2 > 0, X” and Y” are connected through W” and {Y”, X;, . . . , XE} are independent. The first probability in the right-hand side of (4.2) is P ;ixnrun(x”, [
P(x)l,
rate. The
M
P $c dr=xl
log 8
> IL} W
-
{
Q(x)1
0
e.
Proof of Theorem 4 (cont.): According to Lemma 5, it suffices to show that the following expression goes to 0 as n + co, for every p > 0, *..
c
Px-(Cl).*.px-(CM)
c
CMEA”
ClEA*
* 1 log = $5
‘*.
log
c
c PEEP
i C2EAn
W”(Y”
> ~
(Y”)
c
c
. 1 $- exp (ix-w-(cl, {
G,j(yn)
=
v,,j(~“)W’in,j(~“)
i~n~n(Xjn,
(4.4) I
Ml,
(4.5)
Note that for every y” E Bn both {Vn,j(yn)}jNl and {Z,, j (y”)}y=r are independent collections of random variables because {XT}jM_r are independent. According to (4.4) and (4.5), the probability in (4.3) is equal to the expected value with respect to Pyn of
1
Y”)
PX”Y”(Cl>
P[UM(f)
#
UM(yn)]
> 1 + r].
The first term in the right-hand side of (4.6) is equal to 1
Y”))
M
exp (ixnwn(XT, 3=2
p[TM(!/“)
yy”)) > 1+ 27
3=2
$c
1 + T] 5
y/“))
exp (ixnwn(cj,
exp(ix-wn(Xn,
>
+P[TM(f)
M
+ $c
[
exp
I %I
cl~Any”~Bn
+ P
=
j=l
c Px-(4 CMEAn
. . . Px-(cnn)
< P & [
Y”),
K,j(yn)
UdYn) = j$vn,j(yn),
Px-(Cl)
%[C,-CM]w) PY”
...
(4.3)
I
CM 64”
. . . Px-(cM)
c
> pi
PY”(Yn)
j=lcl~A”
.l
+-[~,...&f?
PEB”
c
.
> 1+ r
it is not possible to apply the weak law of large numbers to (4.3) directly because the distribution of each of the random variables in the sum depends on the number of terms in the sum (through n). In order to show that the probability in (4.3) vanishes it will be convenient to condition on Y”, and, accordingly to define the following random variables for every yn E B” and j = l,...,M,
p~-[c,...chl](Yn)
i
Y”))
Had we a maximum over j = 1, . . * , M instead of the sum in (4.3), showing that the corresponding probability vanishes would be a standard step in the random-coding proof of the direct channel coding theorem. In the present case, we need to work harder. Despite the fact that for every j = 1, e. . , M
and
I ~/log
exp (ixnwn(Xy, j=1
[
-< P log
=
) I
which goes to 0 by definition of sup-information second probability in (4.2) is upper bounded by
where
c
Y”)>I(X;Y)+y+$ogr
> 7
1
Y”))
> 1+ r
, I
(4.2)
5 ~w-dY”)
# K,j(Yn)l
j=l = MP[Vn,
l(y”“)
> M],
(4.6)
IEEE TRANSACTIONSON INFORMATIONTHEORY, VOL. 39, NO. 3, MAY 1993
760
O*
whose expectation with respect to Py- yields
&if c
c
px-(zn)pYn(Yn)
ee
x”EA”~“EB~
. l{exp ixnwn(zn, = M
-a 0 l/2
C
C
l/2
y”) > M}
Pxnyn(xn,
1.
y”) exp(--iXnWn(P,
y”))
Fig. 1.
iVf>
which goes to 0 as n + 00 by definition.
Px~y~(~n,yn)l{expix~W~(~n,yln)
> M)
z”EA*~“EB”
= P ;ixnwn(Xn, [
1
+ y )
Y”) > f(X;Y)
which, again, goes to 0 by definition of 7(X; Y). Regarding the second term in (4.6), notice first that
JwM(Y”)l = wn, 1 c
l(Y”)l I YV{K%,
PX-yY”W
l(Y”)
5 w
XnEAn
5 1.
(4.7)
Therefore, using (4.7) and the Chebychev inequality, we get p[TM(Yn)
> 1 + Tl I p[TM(Yn)
- E[TM(Y”)]
> T]
5 -$ var (G&P))
where we have used the fact that {Zn, j(y”)}gr are i.i.d. Finally, unconditioning the expectation on the right side of (4.8), we get $pz,
l(Y”)l
xnEAnyn~Bn pX”Y”(Zn, .(
2
y”)
px-(~“)PY”(LP)
. l{exp ixnwm(z?, = E + [
y”) 5 M}
exp ix-wn(Xn,
+1 $
Y”)
exp ixnwn(Xn,
Y”) 5 1
II
where the expectation in the right-hand side is with respect to Pxnyn and can be decomposed as E
+E
$
exp ixnwn(Xn,
Y”)
.1 A {
exp ixnwn(Xn,
Y”) < exp 7
$
exp ixnw-(Xn,
Y”)
[ . 1 exp 7 {
I exp
< &
exp ixnwn(Xn,
;ix-w7L(x”,
0
W e remark that in most cases of interest in applications, X and W will be such that (X; Y) is information stable in which case, the upper bound in Theorem 4 is equal to the input-output mutual information rate (cf. Lemma 1). B. Converse Resolvability Theorem Theorem 4 together with the converse resolvability theorem proved in this subsection will enable us to find a general formula for the resolvability of a channel. However, let us start by giving a negative answer to the immediate question as to whether the upper bound in Theorem 4 is always tight. Example 1: Consider the 3-input-2-output memoryless channel of Fig. 1, and the i.i.d. input process X that uses 0 and 1 with probability l/2, respectively. It is clear that 7(X; Y) = 1(X; Y) = 1 bit/symbol. However, the deterministic input process that concentrates all the probability in (e,. .. , e) achieves exactly the same output statistics. Thus S(X) = s(X) = 0. On the other hand, it turns out that we can find a capacity-achieving input process for which the bound in Theorem 4 is tight. (We will see in the sequel that this is always true.) Let X’ be the uniform distribution on all sequencesthat contain no symbol e and the same number of O’s and l’s (i.e., their type is (l/2, 0, l/2)). The entropy rate, the resolution rate and the mutual information rate of this process are all equal to 1 bit/symbol. Moreover, any input process which approximates X’ arbitrarily accurately cannot have a lower entropy rate (nor lower resolution rate, a fortiori). To see this, first consider the case when the input is restricted not to use e. Then the input is equal to the output and close variational distance implies that the entropies are also close (cf. Lemma 4). If e is allowed in the input sequences,then the capabilities for approximating X’ do not improve because for any input sequence containing at least one e, the probability that the output sequencehas type (l/2, l/2) is less than l/2. Therefore, the distance between the output distributions is lower bounded by one half the probability of the input sequences containing at least one e. Thus S(X’) = 3(X’) = 1 bit/symbol. The degeneracy illustrated by Example 1 is avoided in important classes of channels such as discrete memoryless channels with full rank (cf. Remark 4). In those settings, sharp results including the tightness of Theorem 4 can be proved using the method of types [8]. In general, however, the converse resolvability theorem does not apply to individual inputs. Theorem 5: For any channel with finite input alphabet and all sufficiently small E > 0,
Y”) 5 1
Y”) > 7(X; Y) + ;
) I
s, > supT(X; Y), X
161.
HAN AND VERDtJ: APPROXIMATION THEORY OF OUTPUT STATISTICS
Proofi The following simple result turns the proof of the converse resolvability theorem into a constructive proof. Lemma 6: Given a channel W with finite input alphabet, fix R > 0, E > 0. If for every y > 0 there exists a collection {Qi}y=r such that
contradicting (4.10). The number of different M/-type distributions on A” is upper bounded by 1 A” j”’ . Therefore, the number of different distributions satisfying (4.13) is upper bounded by N 5 exp (&’
;loglogN>R-y
+ nr)
( A” j=P (nR’+n7)
5 1 A” l=P (nR’+2n7),
(4.9) for sufficiently large n E J, which results in
and rnnyd(Q;W”,
(4.10)
QjnWn) > 2~
; loglogN
R.
Proof of Lemma 6: We first make the point that the achievability in Definition 7 is equivalent to its uniform version where the “sufficiently large n” for which the statement holds is independent of X. To see this, suppose that R is e-achievable in the sense of Definition 7 and denote by n,(e, R, y, W, Xj the minimum n,, such that for all n 2 n, there exists X satisfying (2.4) and (2.5). We claim swno(c, X
R, Y, W, X) < 00.
(4.12)
Assume otherwise. Then there exists a sequence of input processes {Xk}r!r such that KG = n,(E, R, Y, W, Xk)
for sufficiently large n E J, contradicting (4.9) because y > 0 0 is arbitrarily small. Proof of Theorem 5 {cont.): The construction of the N distributions required by Lemma 6 boils down to the construction of a channel derived from the original one and a code for that channel. This is because of Lemma 7, which is akin to the direct identification coding theorem [l] and whose proof readily follows from the argument used in the proof of [7, Theorem 31. Lemma 7: Fix 0 < X < l/2, and a conditional distribution W;: T; -+ T; with arbitrary alphabets T; C A” and T$ c B”. If there exists an (n, M, X) code in the maximal error probability sense for W$, then there exists p > 1 and a where Q: is a distribution on Tg collection { (Qr , Di)}zr and Di c TF, such that for all i = 1,. . . , N,
is an increasing divergent sequence. Construct a new input process X by letting if ?ik 0 such that [D;yl
Px-Y-
(4.21)
> 2a,
if n E I, where I is an infinite set. Define T;(C)
X 5 exp (-nB)+P
= {yn E Bn: (~9, y”) E D$y}
T; =
u
Now, let us choose M so that R1 - 20 5 1 log M < R1 - x9, (4.24) n for arbitrary 19 > 0. Then, owing to (4.19) and the second inequality in (4.24), the second term on the right-hand side of (4.23) vanishes. Lemmas 8 and 9 along with the left inequality in (4.24) provide the (n, M, exp (-n6’)) code required by Lemma 7 for an infinite number of blocklengths n. Then, (4.17) and (4.24) imply that for sufficiently large n
X"ET2
T;
n (T; x T;)
+ TF by
W ,“(Y” I xn)= C0Wn(yln >
if y” E T;(F) if y” @ T?(9).
) xn)/g(xn),
Now, (4.18) follows immediately because if zn E Tg, then a! < g(P) 5 1. In general, if zn E A”, then 0 5 a(~?) 5 1 and Px”y”[D;y]
=
c
YT) 5 k log M + 8 .
(4.23)
T;(C)
TnXY -- D&
1
kix;w;(XG, [
0(x?) = w”(T;(z”) 1 P) T; = (9 E A”: r~(x~) > cx}
and W;:
Proof of Theorem 5 (cont.): Now it remains to show existence of the channel code required by Lemma 7. This is accomplished via a fundamental result in the Shannon theory applied to X$ and WG. I Lemma 9 [4]: For every n, M and d > 0, there exists an (n, M, X) code for W$ whose maximal error probability X satisfies
; log log N 2 ; log M - 8, 2 R1 - 38, which satisfies (4.9) for arbitrary y > 0 because R1 < supx f(X; Y), and 0 > 0 can be chosen arbitrarily close to those boundaries. Finally, to show (4.10) we apply Lemma 7 to get
Px-(C)a(9)
d(Q;W”,
: QjnWn) > 2Q;W”(Di)
- 2QjnWn(Di)
> 2aQ;W,“(Di)
- 2QyW,“(Di)
> 2a(l - A) - 4x,
where the second inequality follows from (4.18) and the third inequality results from (4.15) and (4.16). With an adequate choice of X (guaranteed by Lemma 7) the right side of (4.25) is strictly positive and (4.10) holds, concluding the proof of 0 Theorem 5.
which together with (4.21) implies that Px-El
> Q,
if n E I. In turn, this implies pY$(Yn) =
c
wyyn
PET;
I X”)PXP (z”)
o(x”>Px-
;ix;&n,
Theorems 4 and 5 and S = supx S(X) readily result in the general formula for channel resolvability.
CT;)
I sI12.n(Y”). Now to prove (4.19), note that if (?,
(4.25)
(4.22) yy”) E Tgy, and n E I
Theorem 6: The resolvability of a channel W with finite input alphabet is given by s = sup7(X;
Y).
(4.26)
X
y”) = ; log wn(Yn I x”> c+“>Py,(Y”>
Remark 2: Theorems 4 and 5 actually show the stronger result:
> 1 log WYYln I x:“) n PY$ (Y”)
s, = supF(X;
Y)
(4.27)
X
> ~ix7zu”‘(x”, y”) + ; log Q n 2 > Rz + ; log o
for all sufficiently small e > 0.
> RI
Having derived a general expression for resolvability in Theorem 6, this section shows that for the great majority of channels of interest, resolvability is equal to the Shannon capacity.6 Let us first record the following fact.
V. RESOLVABILITY AND CAPACITY
infinitely often; where we have used (4.22) to derive the second inequality. Finally, by definition of W$ Px;Y,[T;~-YI
= 1.
0
6 We adhere to the conventional definition of channel caDacitv 13. o. 1011.
163
HAN AND VERDti: APPROXIMATION THEORY OF OUTPUT STATISTICS
Theorem 7: For any channel with finite input alphabet c I s.
(5.1)
Proof: The result follows from Theorem 6 and the following chain of inequalities C < liminf,,,
sup il(xn; in n
Y”)
sup A (X”; X” n
Y”)
5 limsup,,, < sup7(X;
Y).
The first inequality is the general (weak) converse to the channel coding theorem, which can be proved using the Fano inequality (cf. [16, Theorem 11). So only the third inequality needs to be proved, i.e., for all y > 0 and all sufficiently large n, Y”) 5 supqx;
1. P ,2x-w-(X”, [
1
Y”)>c+s
>a,
for all n in an infinite set J of integers. Under such an assumption, we will construct an (n, M, 1 - 01/3) code with (5.2)
X
sup 5(X”; xn n
Proof of Lemma 10: Arguing by contradiction, assume that there exists S > 0, cy > 0 and X” such that
for every sufficiently large n E J. The construction will follow from the standard Shannon random selection with a suitably chosen input distribution. Codebook: Each codeword is selected independently and randomly with distribution Pp(z?)
Y) + 7,
In order to investigate under what conditions equality holds in (5.1) we will need the following definition. Definition 10: A channel W with capacity C is said7 to satisfy the strong converse if for every y > 0 and every sequence of (n, M, X,) codes that satisfy
if a? E G: otherwise,
where G=
X
but this follows from Lemma Al (whose proof hinges on the finiteness of the input alphabet) as developed in the proof of 0 Lemma 1 (cf. Appendix).
~~(a”)/Pp(G), ,
=
-t
xn~An:
W”(D(z%)
1 x”) 2 ;}
Lxqr(x”, n Decoder: The following codeword c; :
y/“) > c + 6
decoding set Di corresponds to
Di = D(q)
- 6 D(q). 3=1 3fZ
Error Probability: 1 log M > C + y, n
W”(D;
1 ci) 5 W”(D”(c;)
it holds that Al + 1,
1 ci) + FW-(D(c) 3=1 1#1
< 1 - ; + FW”(D(c) 1=1 j#,
as n~oo.
Theorem 8: For any channel with finite input alphabet which satisfies the strong converse:
1 ci)
) ci).
(5.3)
Let us now estimate the average of the last term in (5.3) with respect to the choice of ci: c
WV(q)
I c;)px+i)
C,EG
Proof: In view of Theorem 7 and its proof, it is enough to show S < C. The following lemma implies the desired result c > supI(X;
= c
Y) = s.
I j+q
Lemma 10: A channel W that satisfies the strong converse has the property that for all S > 0 and X
1
Y”) > C + 6 = 0.
7This is the strong converse in the sense of Wolfowitz [20].
1 ci)m,
c
X”
X
lim P kix%pv-(X”, n-00 [
W”(D(q)
C,EG
WV%)
I x”)Px-(x”)
XnEA”
= P~@(cj))/Px4G), where py"(D(cj))
=
c
PY-(Y~)
ynEBn
. l{Py-(yy”) < exp (-nC
< exp (-nC
- nS).
- nS)W”(y”
1 cj)}
IEEE TRANSACTIONSON INFORMATIONTHEORY, VOL. 39, NO. 3, MAY 1993
lb4
Fig. 2.
Channel whose capacity is less than its resolvability. 3.’ 65
Thus, the average of the right side of (5.3) is upper bounded by 1 - G + (A4 - 1) exp(-nC
Fig. 3. Information spectrum (probability mass function of the normalized information density) of channel in Example 2 with 61 = 0.1, 62 = 0.15, a = 0.5, n = 1000.
- &)/E+(G)
5 1 - % + exp where using (5.2). Finally, all that remains is to lower bound Px-(G). We notice that Pxn(G) = P[Z 2 a/2], with 2 = Wn(D(Xn) ( Xn). The random variable 2 lies in the interval [0, l] and its expectation is bounded by
[
P 2 2 ;
1
+ ; 2 E[Z]
1
1. = P ,zxnwn(X”, [
Y”) > c + a!
> 01.
Therefore, the right side of (5.4) is upper bounded by 1-t+exp
-?
nS
2 ; for all sufficiently large n E J, thereby completing the construction and analysis of the sought-after code. 0 The condition of Theorem 8 is not only sz@cient but also necessary for 5’ = C. This fact along with a general formula for the Shannon capacity of a channel (obtained without any assumptions such as information stability): c = supI(X;
Y)
I Zlr...rG)
= q-JwYi i=l
0 < & < l/2,
I Xi) + (1 - N,l-Iwz(Yi i=l
I Xi)
i = 1, 2.
i log min {a, 1 - a} + max {u, V} 5 L log Q exp nu + (l-
o) exp nv)
v}.
Therefore, the information spectrum evaluated with the optimal i.i.d. input distribution X (which assigns equal probability to all the input sequences) converges asymptotically to the distribution of
where we have identified u and v the maximum in (5.6). If (Xj, Yj) (which occurs with probability a), the first term in (5.6) exceeds that wx,w,
n
# x:),
A typical information spectrum for this channel and large n is depicted in Fig. 3. The e-capacity of this channel depends on E [19] and its capacity is equal to min {Cl, Cz} where Ci = log 2 - h(Si) is the capacity of BSC;. In order to compute the resolvability of this channel, note first that if the distribution of the random variable A, is a mixture of two distributions, then the lim sup in probability of {A,} is equal to the maximum of the corresponding lim sup in probability that would result if A, were distributed under each of the individual distributions comprising the mixture. Now, to investigate the asymptotic*behavior of the information spectrum, consider the bounds
5 Lx{u:
(cf. the dual expression (4.26) for the channel resolvability) is proved in [17], by way of a new approach to the converse of the channel coding theorem. Important classes of channels (such as those with finite memory) are known to satisfy the strong converse [5], [18], [20]. The archetypical example of a channel that does not satisfy the strong converse is: Example 2: Consider the channel in Fig. 2 where the switch selects one of the binary symmetric channels (BSC) with probability (a, 1 - a), 0 < Q < 1 and remains fixed for the whole duration of the transmission. Thus, its conditional probabilities are 72
= x} f&lb
and
(5.5)
X
W ”(!/l,...,Yn
Wi(Y I x> = (1 - &)l{Y
(Xj, Yj)l = qxj;
with the quantities within are connected through IVr then the expected value of of the second one by
Yj)
= -@X,W(Xj,
ydl +
m47lll~2
I Xj),
(5.7)
lb5
HAN AND VERDir: APPROXIMATION THEORY OF OUTPUT STATISTICS
where the expectations in (5.7) are with respect to the joint distribution of (Xj, Yj) connected through Wi. Reversing the roles of channels 1 and 2, we obtain an analogous expression to (5.7). Therefore, the weak law of large numbers results in
Proof: If R is a (Xi, &)-achievable ID rate, then for every y > 0 and for all sufficiently large n, there exist (n, N, X1, &)-ID codes {(Qr, Di), i = 1,. . . , N} whose rate satisfies ; log log N > R - y.
7(X; Y) = max{Ci,
Ca}.
0
We have seen that for the majority of channels, resolvability is equal to capacity, and therefore the body of results in information theory devoted to the maximization of mutual information is directly applicable to the calculation of resolvability for these channels. Example 2 has illustrated the computation of resolvability using the formula in Theorem 6 in a case where the capacity is strictly smaller than resolvability. For channels that do not satisfy the strong converse it is of interest to develop tools for the maximization of the supinformation rate (resolvability) and of the inf-information rate (capacity). It turns out [17] that the basic results on mutual information which are the key to its maximization, such as the data-processing lemma and the optimality of independent inputs for memoryless systems are inherited by 1(X; Y) and ax; 0
From such a sequence of codebooks { Qy , i = 1, . . . , N} where N grows monotonically (doubly exponentially) with n, we can construct the sequence {Qi = (Qi, Qz, .. .)}E”=, required in Lemma 6, with an arbitrary choice of Qy if i > N. Then {Qi}~!i satisfies (4.9). Furthermore, for i # j and i < N, j 5 N, then for all sufficiently large n, d(QaW”,
QyWn)
> 2Q~W’“(Di)
- aQ,“w”(Di)
> 2(1 - Xl) - 2x2 > 26, satisfying (4.10). Thus, the conclusion of Lemma 6 is that
0
and Theorem 9 is proved.
Theorem 9 and Theorem 4 imply that the (Xi, &)-ID capacity is upper bounded by
VI. RESOLVABILITYAND IDENTIFICATIONVIA CHANNELS A major recent achievement in the Shannon Theory was the identification (ID) coding theorem of Ahlswede and Dueck [l]. The ID capacity of a channel is the maximal iterated logarithm of the number of messages per channel use which can be reliably transmitted when the receiver is only interested in deciding whether a specific message is equal to the transmitted message. The direct part of the ID coding theorem states that the ID-capacity of any channel is lower bounded by its capacity [l]. A version of the converse theorem (soft converse) which requires the error probabilities to vanish exponentially fast and applies to discrete memoryless channels was proved in [l]. The strong converse to the ID coding theorem for discrete memoryless channels was proved in [7]. Both proofs (of the soft-converse and the strong converse) are nonelementary and crucially rely on the assumption that the channel is discrete and memoryless. The purpose of this section is to provide a version of the strong converse to the ID coding theorem which not only holds in wide generality, but follows immediately from the direct part of the resolvability theorem. The link between the theories of approximation of output statistics and identification via channels is not accidental. We have already seen that the proof of the converse resolvability theorem (Theorem 5) uses Lemma 7, which is, in essence, the central tool in the proof of the direct ID coding theorem. The root of the interplay between both bodies of results is the following simple theorem.* Theorem 9: Let the channel have finite input alphabet. Its (Xi, &)-ID capacity is upper bounded by its e-resolvability S,, with 0 < E < 1 - Xr - X2. s We refer the reader to [l], [7] for the pertinent definitions in identification via channels.
supT(X; X
Y),
which is equal to the Shannon capacity under the mild sufficient condition (strong converse) found in Section V. This gives a very general version of the strong converse to the identification coding theorem, which applies to any finite-input channel-well beyond the discrete memoryless channels for which it was already known to hold [7, Theorem 21. It should be noted that Theorem 9 and [7, Theorem l] imply that if 0 < X < Xi, X < X2, E > 0, and E + Xi + X2 5 1, then
where CA is the X-capacity of the channel in the maximal error probability sense and Dxl, xZ is the (X1, &)-ID capacity. Note that unlike the bound on e-resolvability in Theorem 5, (6.1) can be used with arbitrary 0 < 6 < 1, but may not be tight if the channel does not satisfy the strong converse. If the strong converse is satisfied, however, (6.1) holds with equality for all sufficiently small E > 0, because of (4.27) and Theorem 8 as well as the fact that C = CA for all ‘0 < X < 1 due to the assumed strong converse. Consequently, we have the following corollary. Corollary: For any finite-input channel satisfying the strong converse: C=%,xz
=s,
(f-54
if Xi + X2 < 1. The first equality in (6.2) had been proved in [7, Theorem 21 for the specific case of discrete memoryless channels using a different approach.
IEEE TRANSACTIONSON INFORMATIONTHEORY, VOL. 39, NO. 3, MAY 1993
766
we obtain
VII. MEAN-RESOLVABILITYTHEOREMS This section briefly investigates the effect of replacing the worst-case randomness measure (resolution) by the average randomness measure (entropy) on the results obtained in Section IV. The treatment here will not be as thorough as that in Section IV, and in particular, we will leave the proof of a general formula for mean-resolvability for future work. Instead, we will present some evidence that, in channels of interest, mean-resolvability is also equal to capacity. An immediate consequence of (2.8) is that the direct resolvability theorem (Theorem 4) holds verbatim for meanresolvability, i.e., for all E > 0 and X S’,(X) 0, 0 c - y.
(7.5)
Let X” be uniformly distributed on the codewords of the (n, M, X) code. Thus, H(Xn)
= log 111.
2 c - y - [C - p +-y] - x log 2 - i/Z(X) = p/4 - $A) (7.10)
where the first inequality is a result of the Fano inequality, the second inequality follows from (7.5), (7.6), and (7.8) and the last inequality holds for all sufficiently large n. Now applying Lemma 4 to the present case (I R I= an), (7.7) results in IH(Y”)
- H(P)1
Remark 3: The only feature of the BSC used in theproof of Theorem 10 is that H(Y 1 X = Z) is independent of 5, which holds for any “additive-noise” discrete-memoryless channel. A converse mean-resolvability theorem which, unlike Theorem 5, does not hinge on the assumption of finite input alphabet can be proved by curtailing the freedom of choice of the approximating input distribution. In Example 1, we illustrated the pathological behavior that may arise when the approximating input distribution puts mass in sequenceswhich have zero probability under the original distribution. One way to avoid this behavior is to restrict the approximating input distribution to be absolutely continuous with respect to the original input distribution. Theorem 11 (Mean-Resolvability Semi-Converse): For any channel W with capacity C there exists an input process X such that if 2 satisfies lim d(Y”, P) n-03 and P*-
= 0
(7.12)
0, 5qe) n
< 0
5 n0 log 2 + 0 log l/19,
which contradicts (7.10) because 0 can be chosen arbitrarily 0 small.
(7.6)
According to (7.4), for any 0 < 0 < l/2, there exists Xn such that the outputs to Xn and Xn satisfy d(Yn, P)
- +(Xn)
2 7, (7.2)
c.
1 Y”)
- ; log M - ;h(X)
(7.1)
Therefore, in this section our attention can be focused on converse mean-resolvability theorems. First, we illustrate a proof technique which is completely different from that of the converse resolvability theorem (Theorem 5) in order to find the mean-resolvability of binary symmetric channels (BSC).
s>
1 Y”) + &P
> ;H(Xn)
and if the channel satisfies the strong converse, then s 5 c.
- +(X”)
2 c - p
(7.13)
(7.7) infinitely often.
and $2’“)
< c-p+$.
(7.8)
Proof: Let us suppose that the result is not true and therefore there exists ~0 > 0 such that for every input process X we can find 2 such that P+, < Pxn, (7.12) and
Since by definition of BSC H(Y”
1 P)
= H(Y”
1 X”),
$(X”) (7.9)
5 c - ,urJ
(7.14)
HAN AND VERDtl: APPROXIMATION
767
THEORY OF OUTPUT STATISTICS
for some S > 0 because of (7.15) and (7.18). Combining (7.21) and (7.22), we get
are satisfied. Fix 0 < y < ~0 and choose T
2P+%(B) - 2Pp(B) (7.16)
1
For all sufficiently large n, select an (n, M, A) code {(G, W>g”=, (in th e maximal error probability sense) with rate ; log M 2 c - y. Let X” be equal to ci with probability l/M, i = 1, +. . , M. The restriction P+ < Pp means that the approximating distribution can only put mass on the codewords { ci , . +. , CM}. However, the mass on those points is not restricted in any way (e.g., P+ need not have finite resolution). Define the set of the most likely codewords under P+
> &(l
- A) - 2X - 2 exp (-&A)
which is bounded away from zero because of (7.16).
In connection with Theorem 11, it should be pointed out that the conclusion of the achievability result in Theorem 4 holds even if Xn is restricted to be absolutely continuous’ with respect to X” as long as X” is a discrete distribution.’ (Recall that in the proof of Theorem 4, Xn is generated by random selection from Xn.) Remark 4: Recall that a general formula for the individual resolvability of input processes is attainable only for channels that avoid the pathological behavior of Example 1. In [8], it is shown that
S(X) = 1(X; Y) T, = {z~ E A”:
P+(zn)
2
exp
(-n(c
-
Po)(~
f 7”1 (7.17) ’ //J
1
PO)(~
(7.18)
+ ~1).
(Xa>]
2 n(C - Po)(l + 4pj+
VIII. (7.19)
7).
We will lower bound the variational distance between Y” and Y-n by estimating the respective probabilities of the set
iEI
where 1 = {i E {l,...,M}: (7.20) are disjoint, we have
ci E T7}. Since the sets in
2 CW”(B iEI
DIVERGENCE-GAUGED APPROXIMATION
So far, we have studied the approximation of output statistics under the criterion of vanishing variational distance. Here, we will consider the effect of replacing the variational distance d(Y”, Y”) by the normalized divergence:
(7.20)
B = UDi,
P+(B)
if the channel W is discrete memoryless with full rank, i.e., the transition vectors {W(. ] u)}~~A are linearly independent. This class of channels includes as a special case the BSC (Theorem 10). Even in this special case, however, the complete characterization of s(X) remains unsolved.
(T”)
or the lower bound P*,z (TT) L 7/(1+
(7.24)
s=z=c,
From (7.14), we have n(C - PO) 2 E[log l/P*,
(7.23)
along with
whose cardinality is obviously bounded by I Z I< exp (n(C -
0
I Ci)P&i)
We point out that this criterion is neither weaker nor stronger than the previous one. Although we do not attempt to give a comprehensive body of results based on this criterion, we will show several achievability and converse results by proof techniques that differ substantially from the ones that have appeared in the previous sections. We give first an achievability result within the context of information stable input/outputs. Theorem 12: Suppose that (X, Y) is information stable and I(X; Y) < 00, where Y is the output of W due to X. Then,
;I(xi”,...;x$; F-n) On the other hand, Pp(B)
=
= &JW”(” iEI
I Ci) + $W”(O
) c;)
@I
-6}] ;A,
5 -“)1
Py
= I(R”,
x;,
5 $(X”;
‘. . ) xgf,; P)
=I(X;,-,Xgf;P’n) + I(P;
ly
Y”)p[;A,,
- ;E[ix-&Xn, 1 x;,
5 -63 Y”)l{ixn~“(X~,
y”)
’* * ,x2,>
(8.5)
(8.2) where the second term on the right side is less than or equal to log M. This shows that the term o(1) in (8.1) is nonnegative. It remains to upper bound the left side of (8.1): I(?,
x;,
. ‘. )X$,; 5-y
< o}],
where we have used E[A,] = 0 and the first term vanishes asymptotically because (X, Y) is information stable with finite mutual information rate, whereas the expectation of the negative part of an information density cannot be less than -e-i log e (e.g., [14, (2.3.2)]). Thus, the second term in (8.5) also vanishes asymptotically. In the remaining case, +
~“EB”cIEA”
CM EA”
. cW-(f i=l =
c
Y”) - ; log M
1 S) log
**.
c
ClEA”
I
= 0
and we can choose any arbitrarily small 6 > 0 while satisfying M 2 exp (1(X”; Y”) - n6). Now, normalizing by n we can further upper bound (8.3) with
PX-(Cl)**-PX-(CM)
CMEA”
iE[log
.log
; (
exp ix-w-
< E log 1 + $ [ (
where the inequalities follow from log (1 + exp t) 5 log 2 + tl{t > 0) and (8.5), respectively. Now, the theorem follows 0 since 6 can be chosen arbitrarily small.
(Cl, Y/“>
exp ixnwn(Xn,
Y”)
I
,
(8.3)
where the first inequality follows from the concavity of the logarithm and the second is a result of E[exp ixn~n(Xy,
(1 + exp (A, + nS))]
y/“)] = 1,
for all y” E En and j = 1,. +. , M. Consider first the case where M < exp (l(Xn; Yn) - ns) for some S > 0. Using (8.3) and denoting A, = ix-w-(Xn, Yn) - l(Xn; Y”) we
Theorem 12 evaluates the mutual information between the channel output and a collection of random codewords such as those used in the random coding proofs of the direct part of the channel coding theorem. However, the rationale for its inclusion here is the following corollary. The special case of this corollary for i.i.d. inputs and discrete memoryless channels is [21, Theorem 6.31, proved using a different approach. Corollary: For every X such that (X, Y) is information stable, and for all y > 0, E > 0, there exists x whose resolution satisfies $2’“)
< I(X;
Y) + y
HAN AND VERDlj: APPROXIMATION THEORY OF OUTPUT STATISTICS
and
769
for all xn E A”. Thus, for every distribution Xn, 0 5 I(T;
for all sufficiently large n. Proof: The link between Theorem 12 and this section is the following identity
T)
- 1(X”;
P’“)
= D(W”llTn
1X”)
- ll(wyF’”
1 P)
= D(WllY”
1 P)
- D(W”llP”
) P)
= D(Pypq where the second equation follows from (8.9). Thus,
qPyx;,
* ’* , ~&lllY”
I x,“, . . . >xzf,) = 1(X?,.
. . ) XL,; P),
where Y” [Xl”, . . . ,X&l is defined in (4.1). As in the proof of Theorem 4, (8.1) implies that there exists (CT, +. . , cb) such that the output distribution due to a uniform distribution on (CT,+.. , c%) approximates the true output distribution in the sense that their unconditional divergence per symbol can be made arbitrarily small by choosing (l/n) log A4 to be appropriately close to (l/n)l(X”; Y”). 0 A sharper achievability result (parallel to Theorem 4) whereby the assumption of (X, Y) being information stable is dropped and 1(X; Y) is replaced by 7(X; Y) can be shown by (1) letting M = exp (nI(X; Y) + TM?),(2) using log (1 + exp t) 5 log 2 + tl{t > 0} to bound the right side of (8.3), and (3) invoking Lemma Al (under the assumption that the input alphabet is finite). The extension of the general converse resolvability results (Theorems 5 and 9) to the divergence-gauged approximation criterion is an open problem. On the other hand, the analogous exercise with the converse mean-resolvability results of Section VII is comparatively easy. A much more general class of channels than the BSCs of Theorem 10 is the scope of the next result. Theorem 13: Let a finite-input channel W with capacity C be such that for each sufficiently large n, there exists r for which I(T;
7”)
= m&x1(X”;
Y”)
(8.6)
from where the result follows because of (8.8) and the channel capacity converse. C < liminf,,,
’ -n. > --n ;w y 1.
Remark 5: It is obvious that the channel in Example 1 does not satisfy the sufficient condition in Theorem 13, whereas the full-rank discrete memoryless channels (cf. Remark 4) always satisfy that condition. A counterpart of Theorem 11 (mean-resolvability semiconverse) with divergence in lieu of variational distance is easy to prove using the same idea as in the proof of Theorem 13. Theorem 14: For any finite-input channel with capacity C, there exists an input process x such that if X satisfies
and Pkn 0,
‘for all xn E A”.
63.7)
If X is such that = 0
(8.8)
= H(P
T).
1 P’“) - D(~‘nIIT)
+ D(W”llT 2 1(X”;
then
) rr”)
Thus, H(X”)
lim -5ZJ(Y”]]yn) n-+00 n
> c.
Proof: Let ri” maximize l(Xn; Y”). It follows from Lemma 11 and the assumed condition P+, < Pj~n that
and P[T
0
1X”)
Y”) - D(iqFn)
from where the result follows immediately. liminf,,, Proof: The following result will be used. Lemma ll(3, p. 1471: If 1(7?;m = maxxn I(Xn; then, qr;
T)
2 D(WQ(.
Y”),
) x”)IIY”)
with equality for all Z? E A” such that Pyn(z”) > 0. According to Lemma 11 and the assumption in Theorem 13,
I@“;7”) = D(W(.1xyq,
(8.9)
Let us now state our concluding result which falls outside the theory of resolvability but is still within the boundaries of the approximation theory of output statistics. It is a folktheorem in information theory whose proof is intimately related to the arguments used in this section. Fix a codebook {CT, . . . , c&}. If all the codewords are equally likely, the distributions of the input and output of the channel are 1 if xn = c” P&?) = ;,
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 39, NO. 3, MAY 1993
770
and P&P)
= ~gvyYn
1 CT),
(8.10)
j=1
respectively. The issue we want to address, in the spirit of this paper, is the relationship between Yn and the output distribution corresponding to the input that maximizes the mutual information. It is widely believed that if the code is good [with rate close to capacity and low error probability), then Y” must approximate the output distribution due to the input maximizing the mutual information. (See, e.g., [2, section 8.101 for a discussion on the plausibility of this statement.) To focus ideas take a DMC with capacity -C = 1(X; Y) = m;xI(X; Y). Is it true that the output due to a good code looks i.i.d. with distribution Py? Sometimes this is erroneously taken for granted based on some sort of “random coding” reasoning. However, recall that the objective is to analyze the behavior of the output due to any individual good code, rather than to average the output statistics over a hypothetical random choice of codebooks. Our formalization of the folk-theorem is very general, and only rules out channels whose capacity is not obtained through the maximization of mutual information. Naturally, the result ceases to be meaningful for those channels. Theorem 15: For any channel W with finite input alphabet and capacity C that satisfies the strong converse, the following holds. Fix any y > 0 and any sequence of (n, M, X,) codes such that
and ilogM>C-; n Then,” (8.11) for all sufficiently large n, where Y-n is the output due to the (n, M, X,) code (cf. (8.10)) and 7 is the output due to x=” that satisfies I(T; Proofi
Fn) = mxyl(Xn;
Yn
For every Xn, we write
“It can be shown that the output distribution due to a maximal mutual information input is unique.
where the inequality follows from Lemma 11. If we now particularize Xn to the uniform distribution on the (nL M, X,) codebook of the above statement, then (l/n)l(X”; Y”) must approach capacity because of the Fano inequality: $2”;
P)
> (1 - An);
log $l-
2 (1 - X,)(C
- ;)
; log 2 - ; log 2. (8.13)
~~~~ (8.12) and (8.13) result in +yP)
5 $xn,
B”) - (1 - X,)
.(c-$)+;lo,, 5 Y, for sufficiently large n because X, --) 0 and the strongconverse assumption guarantees that the inequalities in (5.1) actually reduce to identities owing to Theorem 8. 0 As a simple exercise, we may particularize Theorem 15 to a BSC, in which case, the output y due to x’” achieving capacity is given by P&y”) for all yn E (0, l}“.
= 2F,
Then, (8.11) is equivalent to
log 2 - y 5 ;H(En), for an arbitrarily small y > 0 and all sufficiently large n. This implies that the output Yn due to the input distribution Xn of a good codebook must be almost uniformly distributed on (0, l}” (cf. [2, example 2, section 8.101). Can a result in the spirit of Theorem 15 be proved for the input statistics rather than the output statistics? The answer is negative, despite the widespread belief that the statistics of any good code must approximate those that maximize mutual information. To see this, simply consider the normalized entropy of Xn versus that of x”: ;H(XII)
- ;H(Xn)
=
where the last two terms in the right-hand side are each asymptotically close to capacity. However, the term (l/n)H(xn I 7”) does not vanish in general. For example, in the case of a BSC with crossover probability p, (l/n)H(r I y”) = h(P). Despite this negative result concerning the approximation of the input statistics, it is possible in many cases to bootstrap some conclusions on the behavior of input distributions with fixed dimension from Theorem 15. For example, in the case of the BSC (p # l/2), the approximation of the first order input statistics follows from that of the output because of the invertibility of the transition probability matrix. Thus, in a good code for the BSC, every input symbol must be equal
HAN AND VERDtJ: APPROXIMATION
171
THEORY OF OUTPUT STATISTICS
to 0 for roughly half of the codewords. As another example, consider the Gaussian noise channel with constrained input power. The output spectral density is the sum of the input spectrum and the noise spectrum. Thus, a good code must have an input spectrum that approximates asymptotically the water-filling solution. The conventional intuition that regards the statistics of good codes as those that maximize mutual information, constitutes the basis for an important component of the practical value of the Shannon theory. The foregoing discussion points out that that intuition can often be put on a sound footing via the approximation of output statistics, despite the danger inherent in far-reaching statements on the statistics of good codes.
IX. CONCLUSION Aside from the setting of system simulation alluded to in the introduction, we have not dwelled on other plausible applications of the approximation theory of output statistics. Rather, our focus has been on highlighting the new information theoretic concepts and their strong relationships with source coding, channel coding and identification via channels. Other applications could be found in topics such as transmission without decoding and remote artificial synthesis of images, speech and other signals (e.g., [12]). A novel aspect of our development has been the unveiling of sup/inf-information rate and sup-entropy rate as the right way to generalize the conventional average quantities (mutual information rate and entropy rate) when dealing with nonergodic channels and sources. We have seen that those concepts actually open the way towards new types of general formulas in source coding (Section III), channel coding [17] and approximation of output statistics (Section IV). In particular, the formula (5.5) for channel capacity [17] exhibits a nice duality with the formula for resolvability (4.26). In parallel with well-established results on channel capacity, it is relatively straightforward to generalize the results in this paper so as to incorporate input constraints, i.e., cases where the input distributions can be chosen only within a class that satisfies a specified constraint on the expectation of a certain cost functional. Presently, exact results on the resolvability of individual input processes can be attained only within restricted contexts, such as that of full-rank discrete memoryless channels [8]. In those cases, the resolvability of individual inputs is given by the sup-information rate; this provides one of those rare instances where an operational characterization of the mutual information rate (for information stable input/output pairs) is known. Whereas our proof of the achievability part of the resolvability theorem holds in complete generality, the main weakness of our present proof of the converse part is its strong reliance on the finiteness of the input alphabet. So far, we have not mentioned how to relax such a sufficient condition. However, it is indeed possible to remove such a restriction for a class of channels. In a forthcoming paper, the counterpart of Theorem 5 will be shown for infinite-input channels under a mild smoothness condition, which is satisfied, for example, by additive Gaussian noise channels with power constraints.
APPENDIX In this appendix, we address a technical issue dealing with the information stability of input-output pairs. Lemma Al: Let G > log I A 1, then, 1 ,ixnwn(X”,
Y”)l
Y”) > G
;i&X”,
+ 0.
Proof: The main idea is that an input-output pair can attain a large information density only if the input has low probability. Since for all (9, y”) E A” x B” ixnWn(xn,
y”) I log
1 P&I?)
(A.11
we can upper bound
where DG = (9
E A”: Pp(z”)
5 exp(-nG)}
(A.31
The right side of (A.2) can be decomposed as E ;l{Xn -
E DG} log
APx~(DC)
log
px- (DG) px- (xn)
1
Px-(DG)
5 Px- (DG) i log ( DG I -i [
log Px~
(DG)
1
(A.4)
because entropy is maximized by the uniform distribution. Now, bounding I DG 11 I A In and Px-(DG)
0, the result follows. One consequence of Lemma A.1 is Lemma 1. Proof of Lemma 1: First, we lower bound mutual information for any y > 0 as il(X”; n
Y”) 2 E ;ixnw-(X”,
Y”)
1
Y”) > 1(X; Y) - y . (A.5)
It is well known [ 141 that the first term in the right side of (A.5) vanishes and the probability in (A.5) goes to 1 by definition of L(X; Y). Thus, 1(X”; Y”)/n > 1(X; Y) - 2y for all sufficiently large n.
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 39, NO. 3, MAY 1993
712
Conversely, we can upper bound mutual information as $X”;
161R.M. Gray and L.D. Davisson, “The ergodic decomposition of sta-
Y”)
5 E ;ixnwn(Xn, [
Yn)l
;ix&X”,
Y”) 2 G
{
+ GP ;ixnwn(X’“, [
+PCX; y>+Yl 1. i ’P nZX”pv”(Xn,
1 1
11
Y”) > 7(X; Y) + y
Y”) gx;Y)+y
1
G4.6)
If G is chosen to satisfy the condition in Lemma Al then (A.6) results in ll(X’“, n
Y”) 5 7(X; Y) + 27,
for all sufficiently large 12.
q
ACKNOWLEDGMENT Discussions with V. Anantharam, A. Barron, M. Burnashev, A. Orlitsky, S. Shamai, A. Wyner, and J. Ziv are acknowledged. References [9], [13], [21] were brought to the authors’ attention by I. Csiszar, D. Neuhoff, and A. Wyner, respectively. Fig. 3 was generated by R. Cheng. REFERENCES
PI R. Ahlswede and G. Dueck, “Identification via channels,” IEEE Trans. Inform. Theory, vol. 35, pp. 15-29, Jan. 1989.
PI T. M. Cover and J. A. Thomas, Elements of Information Theory. New
York: Wiley, 1991. [31 I. Csiszar and J. Komer, Information Theory: Coding Theorems for Discrete Memoryless Systems. New York: Academic, 1981. 141 A. Feinstein, “A new basic theorem of information theory,” IRE Trans. PGIT, vol. 4, pp. 2-22, 1954. “On the coding theorem and its converse for finite-memory [51 -, channels,” Inform. Contr., vol. 2, pp. 25-44, 1959.
tionary discrete random processes,” IEEE Trans. Inform. Theory, vol. IT-20, pp. 625-636, Sept. 1974. [71 T. S. Han and S. Verdti, “New. results in the theory and applications of identification via channels,” IEEE Trans. Inform. Theory, vol. 38, pp. 14-25, Jan. 1992. “Spectrum invariancy under output approximation for full-rank PI -, discrete memoryless channels,” Probl. Peredach. Inform. (in Russian), no. 2, 1993. PI G.D. Hu, “On Shannon theorem and its converse for sequences of communication schemes in the case of abstract random variables,” Trans. Third Prague Conf Inform. Theory, Statistical Decision Functions, Random Processes, Czechoslovak Academy of Sciences, Prague, 1964, pp. 285-333. 11013. C. Kieffer, “Finite-state adaptive block-to-variable length noiseless coding of a nonstationary information source,” IEEE Trans. Information Theory, vol. 35, pp. 1259-1263, 1989. 1111D. E. Knuth and A. C. Yao, “The complexity of random number generation,” in Proceedings of Symposium on New Directions and Recent Results in Algorithms and Complexity. New York: Academic Press, 1976. I121 R. W. Lucky, Silicon Dreams: Information, Man and Machine. New York: St. Martin’s Press, 1989. P31 D.L. Neuhoff and P.C. Shields, “Channel entropy and primitive approximation,” Ann. Probab., vol. 10, pp. 188-198, Feb. 1982. 1141 M. S. Pinsker, Information and Information Stability of Random Sariables and Processes. San Francisco: Holden-Day, 1964. 1151 J.M. Stoyanov, Counterexamples in Probability. New York: Wiley, 1987. P61 S. Verdti, “Multiple-access channels with memory with and without frame-synchronism,” IEEE Trans. Inform. Theory, vol. 35, pp. 605-619, May 1989. [I71 S. Verde and T. S. Han, “A new converse leading to a general channel capacity formula,” to be presented at the IEEE Inform. Theory Workshop, Susono-shi, Japan, June 1993. Psi J. Wolfowitz, “A note on the strong converse of the coding theorem for the general discrete finite-memory channel,” Inform. Contr., vol. 3, pp. 89-93, 1960. “On channels without capacity,” Inform. Contr., vol. 6, pp. 1191-, 49-54, 1963. “Notes on a general strong converse,” Inform. Contr., vol. 12, PO1 -, pp. 14, 1968. 1211A.D. Wyner, “The common information of two dependent random variables,” IEEE Trans. Inform. Theory, vol. IT-21, pp. 163-179, Mar. 1975.
