Improved Hoeffding's Lemma and Hoeffding's Tail Bounds

The purpose of this letter is to improve Hoeffding's lemma and consequently Hoeffding's tail bounds. The improvement pertains to left skewed zero mean random variables $X\in[a,b]$, where $a<0$ and $-a>b$. The proof of Hoeffding's improved lemma uses Taylor's expansion, the convexity of $\exp(sx), s\in {\bf R}$ and an unnoticed observation since Hoeffding's publication in 1963 that for $-a>b$ the maximum of the intermediate function $\tau(1-\tau)$ appearing in Hoeffding's proof is attained at an endpoint rather than at $\tau=0.5$ as in the case $b>-a$. Using Hoeffding's improved lemma we obtain one sided and two sided tail bounds for $P(S_n\ge t)$ and $P(|S_n|\ge t)$, respectively, where $S_n=\sum_{i=1}^nX_i$ and the $X_i\in[a_i,b_i],i=1,...,n$ are independent zero mean random variables (not necessarily identically distributed). It is interesting to note that we could also improve Hoeffding's two sided bound for all $\{X_i: a_i\ne b_i,i=1,...,n\}$. This is so because here the one sided bound should be increased by $P(-S_n\ge t)$, wherein the left skewed intervals become right skewed and vice versa.


I. INTRODUCTION
B Y googling Hoeffding and Signal Processing we obtain many applications and theoretical uses of Hoeffding's bounds to signal processing, e.g., to time series analysis, compressed sensing, sensor networks, financial signal processing, to name but a few. Other applications are to machine learning, communication and information theory [3], randomized algorithms [2], to name but a few. Communication networks are treated in [2] where the authors re derive Hoeffding's results and present applications to packet routing in sparse networks. Our starting point is to present Hoeffding's lemma without proof somewhat differently than the original one and then present the improved Hoeffding's lemma and prove it. Most of the proof is not new and presented here for the sake of completeness. When the underlying distribution is skewed to the left we show how the bound in Hoeffding's Lemma can be improved.
We assume that X is a zero mean real valued random variable such that X ∈ [a, b], a < 0, b > 0. Then, Hoeffding's lemma [1] states that for all s ∈ R, s > 0, Let A and G denote the arithmetic and geometric means of |a| and b, respectively. Noting that b − a = b + |a| we have D. Hertz is retired. e-mail: hertz@013.net.il or drdavidhertz@gmail.com. Manuscript submitted on October 3, 2020 to IEEE Signal Processing Letters. and G := |a|b. ( Hence, The improvement in Hoeffding's bound pertains to the case |a| > b. We have where since A ≥ G gives for −a > b a tighter bound than for −a < b. The proof will be given in the next Section. The organization of the remaining Sections is as follows.
in Section II we present the proof of Hoeffding's improved lemma. In Section III we present Hoeffding's improved one sided tail bound and its proof. In Section IV we present Hoeffding's improved two sided tail bound and its proof. Finally, in Section V we give the conclusion.

II. PROOF OF HOEFFDING'S IMPROVED LEMMA
Since e sx is a convex function of x and s > 0 is a parameter we obtain and Hence, since EX = 0, using and the convexity of exp(sx) and some algebra we obtain where ψ(u) := −λu + ln(1 − λ + λe u ).
Since u > 0 and ψ(u) is well defined, by Taylor's expansion we obtain We have and Let Since Using (14) and the fact u > 0 we obtain that τ ∈ [λ, 1], where the endpoints λ and 1 correspond to u = 0 and u = ∞, respectively. Now, for −a ≤ b since λ ≤ 0.5 the maximum of τ (τ − 1) is attained at τ = 0.5 and is 0.25 giving Hoeffding's original lemma. However, for −a > b , λ > 0.5 the maximum is attained at τ = λ. Using (8) we obtain after some algebra Hence, using (9) we obtain This completes the proof.

III. HOEFFDING'S IMPROVED ONE SIDED TAIL BOUND
Let X 1 , ..., X n be independent random variables such that X i ∈ [a i , b i ], a i < 0, b i > 0 and EX i = 0 for i = 1, ...n. Let S n := n i=1 X i then ES n = 0 . For all s > 0 we have P (S n ≥ t) = P e sSn ≥ e st Chernoff ≤ e −st Ee sSn Markov ′ s inequality = e −st Π n i=1 Ee sXi . X i ′ s are independent.
(18) Now define and Hence, using (18) we obtain where the mixed sum M 2 n is defined by Finally, minimizing the exponent in the last inequality we obtain P (S n ≥ t) ≤ exp(−0.5(t/M n ) 2 ).
Now letM Consequently, we obtain the new bound (28) If we define k by k := t/M n we obtain Hence,M n is a natural unit for X i , i = 1, ..., n and S n that we denote by [X i ] = [S n ] =M n . Using this unit we obtain Further, note that if X i ∈ [a, b], i = 1, ..., n are zero mean independent random variables (not necessarily identically distributed) then using (2) and (3) we obtain and IV. HOEFFDING'S IMPROVED TWO SIDED BOUND We will prove that P (|S n | ≥ t) ≤ exp(−0.5(t/M n ) 2 n) + exp(−0.5(t/N n ) 2 n), (33) whereM 2 n is as defined in (24) andN 2 n is a sort of its complement. I.e.,N 2 n := N 2 n /n, where and I, J are as defined in (19) and (20), respectively. Proof of (33).
Hence, in this case we should useN n instead ofM n and we obtain This completes the proof. Note that unlessM n =N n we can not define k as in the one sided case. It is easily seen and interesting to note that if [a i , b i ] = [a, b], i = 1, ...n, we obtain P (|S n | ≥ t) ≤ exp(−0.5(t/G) 2 n) + exp(−0.5(t/A) 2 n).
(36) Hence, unless A = G we thus improved Hoeffding's two sided tail bound.

V. CONCLUSION
In this letter we presented Hoeffding's improved lemma and Hoeffding's improved one and two sided tail bounds for bounded random variables. The improvement pertains only to intervals [a, b], where −a > b and are associated with the geometric mean of −a and b. For −a < b Hoeffding's lemma and Hoeffding one sided tail bound remain intact and are associated only with the average of −a and b. It is interesting to note that we could also improve Hoeffding's two sided bound for all {X i : a i = b i , i = 1, ..., n}. This is so because here the one sided bound should be increased by the bound for P (−S n ≥ t), wherein left skewed intervals become right skewed and vice versa. Perhaps, further research will focus on trying to improve other inequalities that use Hoeffding's results.