From dc1daed2fca47fc335d6f02994f770a10285058e Mon Sep 17 00:00:00 2001 From: Jason Brooks Date: Thu, 18 Aug 2016 10:05:15 -0700 Subject: [PATCH 1/2] post about atomic system containers and kubernetes --- ...kubernetes-in-containers-on-atomic.html.md | 179 ++++++++++++++++++ source/images/guestbook-ftw.png | Bin 0 -> 35132 bytes 2 files changed, 179 insertions(+) create mode 100644 source/blog/2016-08-18-running-kubernetes-in-containers-on-atomic.html.md create mode 100644 source/images/guestbook-ftw.png diff --git a/source/blog/2016-08-18-running-kubernetes-in-containers-on-atomic.html.md b/source/blog/2016-08-18-running-kubernetes-in-containers-on-atomic.html.md new file mode 100644 index 00000000..7e26c152 --- /dev/null +++ b/source/blog/2016-08-18-running-kubernetes-in-containers-on-atomic.html.md @@ -0,0 +1,179 @@ +--- +title: Running Kubernetes in Containers on Atomic +author: jbrooks +date: 2016-08-18 16:57:25 UTC +tags: atomic, kubernetes, centos, docker +published: false +comments: true +--- + +The [atomic hosts](http://www.projectatomic.io/docs/introduction/) from CentOS and Fedora earn their "atomic" namesake by providing for atomic, image-based system updates via rpm-ostree, and atomic, image-based application updates via docker containers. + +This "system" vs "application" division isn't set in stone, however. There's room for system components to move across from the [somewhat](http://www.projectatomic.io/blog/2016/07/hacking-and-extending-atomic-host/) rigid world of ostree commits to the freer-flowing container side. + +In particular, the key atomic host components involved in orchestrating containers across multiple hosts, such as flannel, etcd and kubernetes, could run instead in containers, making life simpler for those looking to test out newer or different versions of these components, or to swap them out for alternatives. + +READMORE + +[Suraj Deshmukh](https://twitter.com/surajd_) wrote a post recently about [running kubernetes in containers](https://deshmukhsuraj.wordpress.com/2016/07/19/hybrid-setup-for-multi-node-kubernetes/). He wanted to test kubernetes 1.3, for which Fedora packages aren't yet available, so he turned to the upstream [kubernetes-on-docker](https://github.com/kubernetes/kubernetes.github.io/blob/master/docs/getting-started-guides/docker-multinode/master.md). + +Suraj ran into trouble with flannel and etcd, so he ran those from installed rpms. Flannel can be tricky to run as a docker container, because docker's own configs must be modified to use flannel, so there's a bit of a chicken-and-egg situation. + +One solution is [system containers for atomic](http://www.scrivano.org/2016/03/24/system-containers-for-atomic/), which can be run independently from the docker daemon. [Giuseppe Scrivano](https://twitter.com/gscrivano) has built example containers [for flannel](https://hub.docker.com/r/gscrivano/flannel/) and [for etcd](https://hub.docker.com/r/gscrivano/etcd/), and in this post, I'm describing how to use these system containers alongside a containerized kubernetes on an atomic host. + +### setting up flannel and etcd + +You need a very recent version of the `atomic` command. I used a pair of CentOS Atomic Hosts running the ["continuous"](https://wiki.centos.org/SpecialInterestGroup/Atomic/Devel) stream. + +The master host needs etcd and flannel: + +``` +# atomic pull gscrivano/etcd + +# atomic pull gscrivano/flannel + +# atomic install --system gscrivano/etcd +``` + +With etcd running, we can use it to configure flannel: + +``` +# export MASTER_IP=YOUR-MASTER-IP + +# runc exec gscrivano-etcd etcdctl set /atomic.io/network/config '{"Network":"172.17.0.0/16"}' + +# atomic install --name=flannel --set ETCD_ENDPOINTS=http://$MASTER_IP:2379 --system gscrivano/flannel +``` + +The worker node needs flannel as well: + +``` +# export MASTER_IP=YOUR-MASTER-IP + +# atomic pull gscrivano/flannel + +# atomic install --name=flannel --set ETCD_ENDPOINTS=http://$MASTER_IP:2379 --system gscrivano/flannel +``` + +On both the master and the worker, we need to make docker use flannel: + +``` +# echo "/usr/libexec/flannel/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker" | runc exec flannel bash +``` + +Also on both hosts, we need this docker tweak ([because of this](https://github.com/kubernetes/kubernetes/issues/4869)): + +``` +# cp /usr/lib/systemd/system/docker.service /etc/systemd/system/ + +# sed -i s/MountFlags=slave/MountFlags=/g /etc/systemd/system/docker.service + +# systemctl daemon-reload + +# systemctl restart docker +``` + +On both hosts, some context tweaks to make SELinux happy: + +``` +# mkdir -p /var/lib/kubelet/ + +# chcon -R -t svirt_sandbox_file_t /var/lib/kubelet/ + +# chcon -R -t svirt_sandbox_file_t /var/lib/docker/ +``` + +### setting up kube + +With flannel and etcd running in system containers, and with docker configured properly, we can start up kubernetes in containers. I've pulled the following docker run commands from the [docker-multinode](https://github.com/kubernetes/kube-deploy/tree/master/docker-multinode) scripts in the kubernetes project's kube-deploy repository. + +On the master: + +``` +# docker run -d \ + --net=host \ + --pid=host \ + --privileged \ + --restart="unless-stopped" \ + --name kube_kubelet_$(date | md5sum | cut -c-5) \ + -v /sys:/sys:rw \ + -v /var/run:/var/run:rw \ + -v /run:/run:rw \ + -v /var/lib/docker:/var/lib/docker:rw \ + -v /var/lib/kubelet:/var/lib/kubelet:shared \ + -v /var/log/containers:/var/log/containers:rw \ + gcr.io/google_containers/hyperkube-amd64:$(curl -sSL "https://storage.googleapis.com/kubernetes-release/release/stable.txt") \ + /hyperkube kubelet \ + --allow-privileged \ + --api-servers=http://localhost:8080 \ + --config=/etc/kubernetes/manifests-multi \ + --cluster-dns=10.0.0.10 \ + --cluster-domain=cluster.local \ + --hostname-override=${MASTER_IP} \ + --v=2 +``` + +On the worker: + +``` +# export WORKER_IP=YOUR-WORKER-IP + +# docker run -d \ + --net=host \ + --pid=host \ + --privileged \ + --restart="unless-stopped" \ + --name kube_kubelet_$(date | md5sum | cut -c-5) \ + -v /sys:/sys:rw \ + -v /var/run:/var/run:rw \ + -v /run:/run:rw \ + -v /var/lib/docker:/var/lib/docker:rw \ + -v /var/lib/kubelet:/var/lib/kubelet:shared \ + -v /var/log/containers:/var/log/containers:rw \ + gcr.io/google_containers/hyperkube-amd64:$(curl -sSL "https://storage.googleapis.com/kubernetes-release/release/stable.txt") \ + /hyperkube kubelet \ + --allow-privileged \ + --api-servers=http://${MASTER_IP}:8080 \ + --cluster-dns=10.0.0.10 \ + --cluster-domain=cluster.local \ + --hostname-override=${WORKER_IP} \ + --v=2 + +# docker run -d \ + --net=host \ + --privileged \ + --name kube_proxy_$(date | md5sum | cut -c-5) \ + --restart="unless-stopped" \ + gcr.io/google_containers/hyperkube-amd64:$(curl -sSL "https://storage.googleapis.com/kubernetes-release/release/stable.txt") \ + /hyperkube proxy \ + --master=http://${MASTER_IP}:8080 \ + --v=2 +``` + +### get current kubectl + +I usually test things out from the master node, so I'll download the newest stable kubectl binary to there: + +``` +# curl -sSL https://storage.googleapis.com/kubernetes-release/release/$(curl -sSL "https://storage.googleapis.com/kubernetes-release/release/stable.txt")/bin/linux/amd64/kubectl > /usr/local/bin/kubectl + +# chmod +x /usr/local/bin/kubectl +``` + +### test it + +It takes a few minutes for all the containers to get up and running. Once they are, you can start running kubernetes apps. I typically test with the [guestbookgo atomicapp](https://github.com/projectatomic/nulecule-library/tree/master/guestbookgo-atomicapp): + +``` +# atomic run projectatomic/guestbookgo-atomicapp +``` + +Wait a few minutes, until `kubectl get pods` tells you that your guestbook and redis pods are running, and then: + +``` +# kubectl describe service guestbook | grep NodePort +``` + +Visiting the `NodePort` returned above at either my master or worker IP (these kube scripts configure both to serve as workers) gives me this: + +![](images/guestbook-ftw.png) diff --git a/source/images/guestbook-ftw.png b/source/images/guestbook-ftw.png new file mode 100644 index 0000000000000000000000000000000000000000..d542b012cabe598c468f2ca67ab74e917dffe38d GIT binary patch literal 35132 zcmd?QWmJ|?w*~s5q_lLGgdi>5B}htlgLHR?q_mRKNJ)1$N=T!Gbc4i8cisK^opb)& zasJ(3=RU(R9KiGJTx+hm=3YDUouV}QGva3u2n7ADjHC(#@>CH5fty4|1izV2Pmc%x z!MljRRYL}kFS1z#_?gI6O3PK%;k~Pev9meE!rsBooY}?H+1%XT#nQp`@M)(o1VRaU zD=DVtnX$j<;i+o=h>AnCQpYW#EJtSB%~+_dtAp_yiVzs{4Vt~gnaN>SXlCA z1`eGS{{2mIO13u){dl*ul3GM$d5a6}zdzX<&wVOO65YS@pVwXlbw@_P9+@W}aQ^i+ zMa-Xx!Jk>tGjHzrZ!CE_>t}y|@%r7khBoS2bq|h=peH0GY}#8~pKR_dwfi8by-=*2 zk)Zp$+80BTbP-*mGCfl`Vd;3h_CvPfBdqP>6WhfWJl#h7@+0>oyd@K6_mwWVS9(oY z7GqhV;<;Vpxe~{N^x0jDi|r+HrOz!!GsMiyUVQred39qW1fN+a$FS4?9V4oL8N686 z#8+vji+ydvm^XFyi^P~hXYuB$)%vaOtG~ag{+P2}?hpz?BQlxs`gQ@E*Ur*&JiO4v zMBJfe-8xHUV>?dXr0(tyNbJTv@OM}1aRUPbUHNZ6AI;fR8vpo+r>?J)#Fjqdxsf2f za(EawJS@LISC1+kM=mv%B?JedSIYXH#%aOkd*wK=>1;dSfbQH(>UVQiT{n>{fpm7b zynNERd9s$7Q;^wMkm~S%q-GKEXVSI0K+Z+EXjVV`|D4rXrQ{Hfm zrn`M$mknE6>&p2?2O4^MB#7hni8-mTFX3l8g(r|Qt?DP|do$AQ-sccpUqj}*R+{}U z>xpS;k>6B{^8D{_j!sW|e<_KeLsGeH@W1dlU|3jK)LBkE@wtRDyDzW(h?>ZMi__H9 zl;g16A)~06?5m-pLrkZb5wWsj6cQF@@^`Y}$0$7=oypF4E^s7K(CbkzrU(Qge1E3u zdN5Z%TQ3oY){R8r`)=27GvUjZXPTOt>DxFsIPC8GTJT6nG`zfo>wPgy7v#RT2aT4K z`6#c6(TTZjBm8bJv`3RJmXh7Galeh(_Jp7uU0%W_a`cm_Qp+Ff+*=5r)gZdKxY%SS zedPTMTH4j&G|nc+)hGP!`^XTPw{OMV-MM#XDjk}+U%$qEUrQ7j8Tn(5!R{-&?)K26 zFwDxQSOy0NJwdAP+MKTYv)EtX<#mKuSy>VCIlll4rL}Jm5D-wN z--;iOK_)IQA59C75)An|p|!6xVX?Wry*^nW*W`XcoaMO|X;L3e$o?cZH#Z_Oviss- zp51L%O|Quj?yGcMUv|JlWWd8U5+pt$AqcFE@o*}8!4mz2_YZ5PL0fLq!B4cF(B^I1 z4u{@w%y8gMNlcoKe;(qeKUZ&aJSuS5H8q7-QBmQ!o#|2KexQR!!i@%b*J!UloFzo^@bC~T zq!#bE`4g}E#}BE2krC$e76A_^G9)BA8chrfzI$p)o!4zQar17kvXdp6lurcj(%dl+iJ*z$HMB|V)9f>mMIX<%(fJCZG8xHDazo-?y2HI^+BU@&g^ z`iFs>t!)`Zw8`(bA^y`RYF+^WRG011`ZkM;wnp>j$_*Q)a(ysXdV2cRo>0`})6R#R z(v|)FSE=%J>0@kZ^5Tk$u~=ke%y-L=_t{m>o3h=u+^92^h6Dik%-rq?41KBenw_P< zsDNc-udS`Yf9R&nFDO7E;kLz^uF$t>Zm2uGs8fP1a{a$~vYYlH52dg+dfES}F?X+@ zYH~8A5){-si!+P+wH)vm)72$v`I@-Y#phyQtkeGDDaOh zBE>j9S~$Zx$59cXJ^XDr2^|%j*si}llH#ci>A`e~F zru|sN#2@)wwkUacG*8--T9jn9g=^^)zJQ-+YB(YZoAqG$6cQS03`{p7CPwQ0dpbcu z!JJ=zjX2U?%{&W+lxx^{Df#;OVUv=gCCMTcyg~VgOP_>{s=jAg;570Po&f#QZ zaA&rf<^f>8$=4KDeliRS;pkX00Tb|tP5;;pw40n9%I|%BI(}3!z!$5Gd#fClg+s!_ zX;@h?g@uJT8R;ki6L|Y}V0bvBsHkY_8my+Lrza&Xt(5D@BY2<9{3kA{h(9Ut`kvjX zQrg2`d*_h+?LWBuUMDZWltK*suMxz=#JDR_(b1)ReFa_lGtJ4>R8*eV>6DwR=y-ba z0t5&zE@qJb@*2gvog9^bCDGTxZqY(+)rM)$0-uZwJxSL5a=S$38w3|E?ez5YLwY(1 zu-4EwF~o2X6I0XNcMM=t8N34w49J0{PyO^Z*(S64Pq?5bARxH9z6P!{#X;XG6@?E1 z!H1|Qxllogn*MJJ>D*0+9k)jdE@M8+Y#U`J|fdGJXeDA!;|K7V=M_gQ-&1&++3wruqTW%VD{y7fJL4XagK)^XB_@Tq^ zwr6Zi=|s3d1%-!)2NtZfwY3YTJ@%%V(Z~e}&qWJMV%)fCsjbYxS$J$$9e`B=H6W55qxX z*$!?W;aaegfGtQ$OPfxYX%q6;lP1d24!`HTfG$Sm=f8j;pc=y1Nu$H^8+-c-^A7cQ z@34SrQKk${lVEL* zmMzJBDeJ#jhvV&q-S1LOOeT%8E&zz^0S~_aBJ%E(gsZFTzqNk*P0eI`G}H5XGi9R9 z%cZHcbv;3vy!&Kxh}~}Pwd>9}+|tsL=gr@up~dxljnZnb)A!M20{C!naK)-c7&jLO zy8FIhNbiSJg8)*97O3RD{YzCoLb-@lZXJRJuGLlLDRV3s5m4sBCnAcZl}pYs z?T-bFtj&C?ZDYf%%%GjPNGZFoOuKe}4z1Bq0iJz|n-}&0Hul@=^*E80wYA`*BgctT zK!pbe2diz&p8pqrP6(L{{=Bh<1;b-W$xD7j81jGz<-aJn;st6ykKQr{Q~)aMuNBUPUwGWJmSTG zyYxPONelA77xH%H>5%`M;VZx7Rg3#Kg#RJ^3Gn~xFe6^D{rAL;{*w?;f9m>Qvxony z4JACSX(;O%am+Re`1D@HaLRn5^XeK)SxQyI4Ng)rX)q%*^iFriSUw`?B{R``KMeQ} zaLxpp143;%j;A3#{g}IS%v3J!<-5)xD=GeB;s|2V4@*h=4dWe*gE!D`y#t+oWLWTj zZ_PN0cX>ipPa7ONI+65zFhdqbzMkySg4jh2%haq9s5HZeW0omhPCeLY`}oHL1MdPV zs2{US2bS7V1}Qq8r67@q-3HRH&iPPMepdILaMdKN(3k9Msf;31opE?piSnzo7??y! zBrv+-zM-X7gl}@op)YB3*N8`WD7RPaH%J%I-PZAs()dYOgl;tTZe3>xaZD4)s&i-A z6~gUdR;)&+M;aVq{R)gF|IZkW)x{|u$ro9+g$VJUQq%amCDfz~G*}-?v+I(QT%Y(j z$&Oh?g4&{r>_aR^nvu$s6rIKQr?%QohWr{)5 zOvkeY;8I8O4v26}822`Tw<>eNvR1Cx_)>|5x?DCrK2~poRRZM7B7@-DmFfbPBTCJ0 zxfe&lEy^xUXFHer6Kf9qtB9>-NHe>I?^rfAx>`8gG~ehwpecl=W3qu1?=MqZC_p#9r@g< z4(KfoJyNEj+24mzxvndFD){2PWB9pD2fTAi3oMkHY3k&sgquI46D}RUL<0MdikR-w zIO=hH3rF7!{6uDWBEi@OOA||oc>BTLMxQx|1jU!u@WA`nR5>-H1C)7 zd!s%S_ds&ZQ`wVBi0Y!qO_?IGdVmxfC-OT+SCj}>hYyoGJt1J54vibLP=|@ExGPncXNa?CvO{lk*zx#Z{Z7OUA0Anz zx=qP8j&4@pt#|fNYP#@7Bd4M3`DPyetP~Cc?}t%0KO^31fpfx=2?yVO@q>{|kN+|l zG=PwLS+ozo3#7v<- z9rkR}0dOSL?AUE&U5!9}&;ks@Ru6{;7S5R4=;gy_t_X&3=^lM9xL&Y&qTdYAn0L&i zVm&n#M{`ALpweb1L&sPQlV4Zaa)*YMDJ)Z;pYmayHGi5Rt}qZiX-i3HHb!`|RUXpf z6y+?WJMfFdW)$m)nP{Hy&YYhz>HX{3z?p`@UzYt0*!&r^8E;OnfrbHK_mMF z2hq`l`vJZhirqIOt2{33uk8dXIxy+Gc%^E|C_8SIa504>lgd5~&B_z^B2c7q7Z(W^ z|BUx}a4Sn!o!Ra9Rle_Cn11dzhTa&4?}|IyM;1?U&tks||Dqrgc~vj)Iw{e`QmK1P zX2UpqPqYK>?V;kE$%Sth;f2q^%Dq`|iKB8kn2DWH#^93<5ZmXB(+Q<(PDcvvT=+RI z=^AGohl8!|v$g+f=aJvaO;5Xs{P)P5hL{wWW*??p!o}uo18e+H?lU_`a|)}OEUmoN z;6SHKEoFXUyvrHEcLnm`csv2BU%Ya{^NF7#t_Y-iGBRns4GNut7|Y;rAG+ghGN?94 zjHa!OW#z;kGCQSgstPQXBIAo1AJ!^VB>s{ZY3kN+U}O&I-EwivFxT=|l*|zR?Bb8N z$}pBy3|!5&!8+s9m_fNAeKAZJ9&bLs^1DzqEz3^!t>pgcQ#rEaq9e>O4oX6fB;vkA zc*6BOsi(yNT~Sztf^um0Ru`Hz({=*~Lqd4yXZ7H47zl}r(Vm4a^r9C2HX~gBD&>=8 z%uVAqXq)QdvU0y~V_y1|6ht-gpZyM6?L0qM*x&;iuuH;5pgPQwYC<6DE7zwGxM`fp z)t=r9mF9f;BG+nKJ~E#FcO2cov``#pY0>(T$wdRvwNsiB*cAwVFWp%VCyDY|vJg7c zqG+?TTh7QP#y93Humq(Qf(sSs=OrFb(AgH+K4z?UQg09V?~G+KoNK$$Jj`!1Z6680 zXya&^^-|b#3@M9_FKaGCYpkHU?YelxV8A^h6++0Op%W+4>71<(7Dj53@QDYOgs@t$ z(<~_T+(J02N}-?9#i{miH8?QegFY&puHbprSGr!)0j*-<1@<^mbtgZ;F$5Xa{KvB$_JiV8#6RBH$B3d0Aq?K&zkk{r(s?{7xVn` zrmS@ZEIwaVo00Z3WQ5pci)n1v`rp9ugTM%~BF@5xfjf`p;`et@oWcB!aZD5=CL8W<< z^Ad-kB6RZu|MFq@;n`)mIL^EA-mQxP#}GR~#+wde9FzE)A3p9P0xk!d1NW!#Q%>gZ z7{q5tL;maKs)2~@;%)4PYxs=ALu0q4jD)cK=3lagAlq(>N|1ScW7dtIL>z>wL3Q-` z@k8PIQ;jI$f!CvsepgN*6%Qj@jA?IOin=ortToAE0RYMPt_S4h4i?NnLdq5ff#+eu zu+N(Haa1I)|3?S>r>2eTO2~DXNxvD3eevlC%LMCzi5am4Jj<1Vt9e%)9z~LO<-?hB z>9&B!E=n9W&Vjf4+Y;}X)yuwqcXaHsXB;t{!6M6OMfj7e23z|dyRMhJnm@Z?T2F5s z#s}=Rn2`W*n@nb{;;%S5mOiSnb7$txJ-Uz>U#oYt#0x%_@ufYHjtXnt(FM{d$g zDn?Nryw@8i$CIOmw?A~T;i6F_|1wg>kan^4FS4o*#~K}G7}R4gOJ+=YjAm#3v9tj- z0c&Eh8xWGIEOvH{vnGzJQT>Yi_dkUmN_WN^0-FuR7VFgi5Pu%RX4_^ey-!5b93=>$ zA^9#Wkz$XTbnu5O;_^F@dng@h67j>KKNT^2ZG%6nSF8K2^~AM`zk~l1buEe3p?JPl zYR4tn3BN=Rp0fD1Wyej{jm*biyG{KOk&b`4x}f}!uEV73ywiv}fobu9K^9eAl@kYj ziT#j{fmY9!E0i>}j!HlHo3yRHs@?rTdymL$E_zy$ST69vKf6ubChb>wC{huAeP3cR zyUugF%`0i@IxO!&L>GM+VJeuZHZGWA2bc?34vP!fJH}O>b&1kmfVq|)4vaoN+&eUJ zlox-c%eK-6UymZ|)V(>g$c%t0z$j&jkgg%|5U0GHhY<-{qkZDFa6tgpLU^>xRkKGr zp8zUfKZX|BCF!Gh^PC$62Zq(q7J)#ah|j{-ISACHrZ+D1q~s-XDD&n*CC(#?Z&bgP#}UP>1%3QD*y`H1 zo2q*1zIJFu^o{s+J^%Zw+nSyFFWi?B&{g%j97@XL^P>2x>F>d;JMI;88s#sO92xi8 zs-I_ksjuaK&vjXd+%6OGj>k$H8p`>){IGFP`!TZ?!!dqvMQwD7m8-m*pp>nksEd%M z!!kNQfM*L2Rbg}$GjFt{9Zf;<1&lqG2sM9xbJkQIqxM$>F@f)8$&CVIzFla_IVq=$ zM#~l8U2R$3Z_g5;UEJS=M6enutitm7RC5*^h_4^@$`iz7^*gPaCBA7L{Zwi)J@qzi zJLfAZEFV&T3gP9lF?AbG#IC48TuUL1R_!Gtjl9G_{U2J7^%F7cxA^Yj~;;ZWcDW%`Equ)Ad^w)3)gqMvhtN%Qe zPvZ{8fm9x|QeU1aZ;(H#-!0XdnH|nW?CX=j;iA3#UAOmS`v?k)B^9*wr8-itwDOJ` z9=v!g%xBw{%IbxSjv51Mb<*9gkmfxN=r#<5r~9R>%i{Tc@8Cjv7^{*lvSvR4Ix?}o zG}ei7xV`LJEDR2s{>5txn?Ftft<=tm1%AIDFKW;XWJpLE6jLUA*Rd!$Tdi?|?=&r#7w( zMrj#9oBRnpDI;fvGGghVr71imQ;HVTb%l$XEv1Hf~>py01UUV(GdSe zF`fk#U{-$qmbn7brsCS0#$Z3cpH&0lfaS{QbC67Yn2W@yFTrOtf~Lxww^L`wJKVaw zjw<7SSp(xz8x>Rj~kPmUJdWbB_eWChAo4%R}KLuTS@!xlM< z&7%`15j*7Ff-vFEM7HxsaB=F)nVzv6dA1rnYtuduwom! zu8^`xyWG~ZxnhknGU;Sq59?{fhuRcs_Us|k#v6vgrq zdWPu`{^eKQUU2-T0M=!XW9dk}DnUB}6+MXoj-gwz9d*efO#cDym5%h@pErw@INPb0{k{828Y6XSYyoFG0moR2;A6pyoeZelmmY^6Mr?&)#L}llTi>b$PNi zk;;ZA8A5<_=ZrSCN)mL*+3w)>&N$vxrCGf7(tL1kifvU;??mw}92TD)OUNT>`NBCF@3MBuSS+6bXkKArh;w~{ z$9eD+yW2}2iAZMKcukY1`r*4+RuYRV!lHCI?FRc6BZ;3NsyfciyKH)R0nFA3e&2tJ zt9QQK^=(A@Iew$D&EbEC_7c^gG(m0+M|sou-`M=Ruvbz%{<{|7bKd8v+JHr>mH>Hm z91?1aI@5<(W>^erQTp}97U!3U@u-YTQ*sHV$Ej#nThcrD5Uos{MbCQVG;w3UaodXG z$+C?s{{5?wYs7!)h?Xs<&JbV1((K**jIr!n*ES50$;RoBxjgsMuyTV0KAa_-OkrF3 zbpp6bpT|uM1zG#9Zz~C7!ckhiIo_u&`RZl|JnTiEe`-a%k7u{vk=7ngpYMxyfo7FX zWC8J)H32cJH?;l4oW!}erj-(nEv`_6w3hFTcJDdsTDf^Ov`-<6_DX)iokeUj|!ZRi5Z+zq|{6XJHNxoUfKYrQHYT`Emv%v8IFydr{z@5Ow5# z;Wm&_uKD3tHeXou<89kB#u^3Yn&(ENk;@RIEk5(Z#V4vJWfE=!dJ8La7V%TZXNLq) zicOVJ(mzHTWtWw0?d%1ELe~OlGc5eB4qLmd<*`+lxZ?w7$N!`q2H&SEx08AkE>?zqkb{OPkH#DoOP#t#{`aFlMK6T+ma|g& z5(%RcX={I~N6j=@P1l>{Q~hlxuwEdIy%U&5#b=NTJH#s}Q5~~3hz0uFNzdm$(TQ=( z3@)se?%xq7jJ?TwvybRC;3Ia^xArP99T^`#i#INGIKA$HTCW6kO z{sZ%iyo|w>ciJACMLzYxo|i_yx-t6diRdy%BfS)KD`#--6l*@J2$<{dBqoF*#-~BGJU=v)FDf}M17tgaEsy`yU#k~V;$Z3 zbn&pYELNZ%gY{fEt(FM{C|$C#n``NCn}j?Ok0*l^BMj^3({e^k61#I#t!r~GkVLm# zZ8D|D3!EMQxT@Azm`HMU!8uKD#xdcj@u`Ou%lxkPpi|9e>l@qL%42_Ec|7cq1=fMZi{eG4~g!bZsPm1BJxcxXc* zMnwAFO|R9C%Y^RC%~IO~=oa~vT7mOV=P9j56IBR0fsPp&^7pHwg&Bca#r5+(j*91& zSGOE%hNQ+^da}?}1Wf^NitpYGlbS4&KczWyL|n>NhN>pC3cq~E%~UG9zm|T{@u2qC zr3xNSTenmCE{aAfeCm4aVc0wN3t{LJIG2tZ4BYpeupD?8or%_`3v6&$)1ZXphoG5X zm7E%jn|UvU_i?CL;#KbF55=r{;xu-Wjyj;x>@z`UNviY z|0}Wuq?l@ckX4dbkD=Gyrcg+#wGP1NhQ<7%q!#afS9|qWiTZHODt%_)x1u9-0dE%;%2oF7GoF1l!3tbDK90DFBKNcm^0@8BSiDq!vURigWMqgmgT$^5+1Ol7iP7LLwcag8DEk)0hQ6=`p9b5Q zEu~(I5SqCW*#H*{Tj2}J!xO`SrN*hPJBm1CoE07yr!;GY+Y>*tyDGJ_j20`VCQRa| z+loTDXvjhG1z|5~;clXe652iF#Nu+0i|`MWEvAG;t=f6gn49}CC?5*wCC31!t_WMY zim__T=rb$`0nI6F6&wTzSCRek;f@NE0r6d&hZK7;T|Rntdwn_p9w>ozhf>H-R7Ct* zDP5Cu%2Y9_3>|O|Hfi$aqF{P0Owk94Ii=5Ga_fm;(ex$FO6HiPCPwWKkavC;ypLG# zT{Yu?AAUsqqOlr5uID>dS%XNQV`Sb5w6J8KN-B19bj8h0Vb520%lAFdJT#%%t zT=SWuDB9mkcUjB{L}ZR@I@fnj@_Il+R|-fVNiDWK0J?x00~AexyZtM zR^gaomEJAlWXi`#>m%yUwhIQrgZr{7bmgqF&ipjtc(F*MAMc?X`TB-_;=VMZ*nJJA z?N^(PRv@P~)))m)i0o~4GJOQ}1>U#niJQg&v#RBjXWr_;BJ&I_#5uU2)gaV>K;2Gb zAgjI?QqZAxJrHYhJ=8R|!~c|&E1l!t5gtUA}H!O3Oes->}; zHbbn)S*AtQuJ^{g9)1H`ZU*ayfk3K$cZHC==QWz?P3<}&b|vpqw9hkS2J(BNb9ckw zLmSkU$@c7>uogTqK(oYW$bKBCCkL?pEPq4}vEzr1k5v!9eG1?hyWzNahg{C%^NcEM_u+w3`2&g3sEpLY zMmL7TI|WHDR9NP4V8TIld4|0MV(q{4Ft-tC(gM&_$LkK@w{LyA>bJipO-3bo9KUuA zzXUFAocd|aNLDEb98s*UBB(FE8F$@5Z0wN8wV$k+^$QjzxZ#wJ;wf|wcDeenSuyB% zKn+xBN$6jifmZj4tqs4lIh2x?n4FSj+%N8#tE3M(i>QcN$WPFCDk%@eU!0|kb{$=s z6Ph~wA)PJ+Dr(DHIZJRDGf&eiMbN~+N~MHZtB6V^jnr)^l&^v}Cn|WiK~ly&neaJx-KXm1eol8Y!<3^T0{p=5=Px84#nh5UZ?#^{k_gr+9tVc*H@ zaQ@@M01Y?<)IVGP(8G%S#a$n;97;VOA&r;q&M>6*xun|w8ikkUSQLg~Qge&MFMd*% zMT#Jw$JKbB>*V#I3qN6Vs+~G4$*sY`MA>lfq{!m;FxIN0FfbUlI_*Y?V`2mlB4;ef zh)t_Q^kthL3~EXYK(5hV+Ol8`#5+V)o#(?hdIKBJz{wV+hZEC|U%ck1C_;BU5>O;V zNXaNai$Qlas+`I*+qujrz^_eB%4i`2nff_LnD~n?npDz8*yw(AXQW|tr-Cqm_{>3M zV(Ka)q6GDSj+ZmQks$2(e;1zqUp+bor-A=Rr#7KKT{Hjn$O_kBKh9zOcA`px0?qS8 z+qd1C+1dHrqn@tzPc%Yyc~|h52Bi?qX@zhr-R{kR%+>2>{bgQJb_S=a-~ETSZ?}B1 zXWQ)imkgO_3i1mz-!y|0W;l=)M3lgZVZ|q;h2kV^v-vzc7d*T2d{I=mSicR*K7H@5 zd9JQb9pbYR+-9aH90V> zNk_x7%dnOgyEfhY;mYrpIGUIn3*va*C*pZLa#`9WUQ@uvtKa5}u03ZvVWTLS^%Qb- zdqcUqi*EMbN~)3xO&!w}s)dsD99vlmI2d&awQRw2VtRVZjEt34Xjnt}m(@L`EtAT1c{BqYzI9~175r_$oj<(_PLz(CU=*vwcyRAHu5o5zP;7fL>nNz> zRkRhHc8C!@q*|}KoDq?=;}N34@gt{*8`FHCuwQC<`hki9R1e3HOfyZ-Pvk1dx$gb$ zJ}FZE&;ki`QbL1+lr6UPZgE1t_pKit%Rstj6D0QiN3O5m$on2M3*Mf{lk#8CbJW&3 zPFrG;WJf5>yMNuzsn)9X^7&?$(dmlUju_W_Q;se3R!-W#1G_QLw94$yCE@wLYwF1R z*1c-!##_|j*x9>+9#j282L~^Q9~De!s_1*+RJ`L9fu`ARjyWPX7v2y^Rbk?oMSs@Q zf$2?3hAvFZ^BqR-5mbkc@Ra4Oa2Em=nz8seU<(-JLXl#obgP#%^pb~>k#eGAL8fdr zOD1<-TY`0e_>0vUzkjDSI!GqlIi4llf7r|!X#wTsPXKb~S<`U~Pn>utVR}FU!&^xT zg*Y*Mcq+yB3uvyk_YY;7d`SbyXFt-DQ$WCyeYR+`aUn;C_%@mG)ztmbdiZht831KG$%-)vILTNfeJNnvy4_p?K7gi1}Hpa zVY!3r3wMdbrH#SWwXE&D9|Tm~Qw(Rs&Ige7l)E4~JvhRX`!&&ndcMN3jJN?80=owG z&3Zn|Unjz(3)P5kHI-K0Y{0c0Pr5(?Y3QksPb&=fEBeVXU0IM7flpCs8n#LAU482z zFLFmN#fgs3fn>CL{wUclWs}DXW9vE|yZJ^>fdqlM9il{s<%mrquCMUdF!T`d+$#P~ z(?f?Vt8IVZlcn?~Hz?(E7K&b0k^8_t^xbYVj@9x@eC@@rV?({4$J(N80+z)a70If6 z)*B<@_`?JUFxKx@k)z4j3B3@XW(AmY=b5hAcIJZPBvMdFrp$GE-%_63+32wN1Tx+@ z^YqGS%hh#v91-9QBv77CVjc0k%67tM=Umjn%&NB9ZCg0~z`E9Lwrv+x@c0Vi>E+!u z^@HfL(ZS9`zeCu-v&?}Nf^~3@4G*{1_y)qD>!9lWhOXnmo*80v5&bRoVwkt7h4(2$ zTq5{rT@w{%?7#?pn>^D-+y#Xes=THMEPvGynQ;*LwzVHk4}Bi5^zeXm#R_%K>3Ubjxdp#-$a@xNTTv*I`$1q-P$5f_u z0tZ>S8PSbs@%}n1v|Ks=acJh@&FwUE)p#a&)P=&gGY zbXnSRyZ82xp4S-pseXDD;KKz&RlyIrv(3HzG}vS~NCXz>VZzfhQ0~h|e|XSvQ*(>a zWQelHc9JaZd--%^2(RZ^zWJ5O+e&$_xs%ZLVzK*M2&6)<)5=5P0tWx%@syCRYSl;d zUIDiY!RR+MEn$y5ch!zf1`}S22^Or>BFo4tomg}Qbz?fY?~Kll#2|rgx@F;eco*Dm zC=TIo&i6hxP!K;4nRu+&*HQ!P@T#(Y<5r6>xk#S$DSf?8-6H$rwA<0Z+hc+^l4?&G zN?Rb*c|KF&a>?txv_n#3T=wrUvspbGXRGrQ)ynRiX#53yNqHO=gSd4qw6${HiL2C% zY`g8LJ*dxn3yl*Qy9GPZbCYA=`u87+#c7R}-(L>h#KU3sE{;q=xgT3x5g=Gg6_jd)QI}-=~r8YWTHjLg5GVdUeME>XrWdLb$BlDLL&_8OfOIoW>utR;g z;T}WF!HSdy?(bL#Jw#Gz)$IG+xnIGrJ^%EoQXdQ6_Uh%@vJH9TwYSc~454%L)uEb1 z@DnVYlz01nzc)8@@S#`UQ#yB55A+j0ZhU-@kC+m))FoE`p%Q8Gh#oweuWcr=;?DPC zB2A^W!b;%~PA$WPxJFJ2>J=R+iFt~VVhW8IG*z4#?eAPj`1qQ+vsVxIRH@uzN%-I? znDt}p(>=6;RY9l*D}V%vX6j`V0vApckRb3#*Yj65SWGHTf2hLxAgVGbF*A0g<|?VMSDpZ@byd!~})BT{P^ zETDkdKB-h>-uWirx~~MFxocz!;R6Lo;`UG)j>(@?2`+1G_{l52t%v8=Jze@(vNMGq z83Cn(;Wgt4bdS~OW&`EZ3`%g|U?u)5#PL=i0^g36OcSOXe^rxz+m8d_<` z)xd7lYsiZibL4rN6XL;7V7~eT{{usYDyMtMRo>HriMVUx#0xfIpbO_`IU@=r;pd#R=EHm zE4I~3qPkJyyd_7qPZ|9&h_IF2djHV~yirsXjI+j9N(PhNUpLcT4P6bQ0##bdAJlo3 z-Y-sPHhHg8WbpgIf$;QI5+6T)vcbGN7>|I}u(tWAHvuOBK)IEdVOq9w+OzFgG7-0n z&dncE_&y9RqLQm_VsQSj#dcPZ@|{++=F0NCQ-&a%jyCMiqWT$-gZLK*#-6xY@b<*f zL!y=r9S--EIjAm72ih&Rx(l!%NMJ}lZhnDUf|Er8 zPwwv>ZxvsG7>Sf)gx58>H848~5FAj}cmUWZ6w|tyJj5|qXQk?pXZS@&Cua<&oQ$f* zGW=dQtl;*HNiW`fqZM4ej{1Jfr_*%C-CFZ~S*vl15YOg41 z>57a_FN6q>Hfpcj&kJS01%ZM-SV45j)@wz@RGvLN2nZT;g1lJh0%u~24MJ?^=glyP z=P~r|SQvYWCJfpMshSoV-|7ysJY5m>t=eIl*o&&j+TL3aeem<gs_G-(B?2$u=t*V7E^gv=(wD(vy>8Y~&a- zw$+-R>kpC_eY`6}y<7lF*ZH$;6ZO{?F^K6JezET5=`^}2A9%L>qr=EQdBL`f}4Sa(^sQj1I!1cuvMx!sYNX^{N_x{lS&?fkl_&53w5>}z`d))wr zUmb57KxV>=-~Na{S8tCIDjo}V7N8VJ^iBG@S}Pb=-Keknc|5`FAW0Oylh@{;pH@gJpEf z0nHFr;Y+oGX)#H)G&Au*vU5=WyC0wU>0|ODVi(|N15k7fq_~o^rGCSeT09YxwOMSf zvj<&ueZ|nZ>Pt=e)C;;NcfL-?J1h)I3qc_f#e%SUW=Ra~b|(^W6}M-;z9b z!<4j?d8eMDn#=95{6V7sAw=cd_wUs-B`u^twh8JC2BpER=tMk3GmjnSX(YHy6?gzT z8$XshM@QAdsI8hC5gr{a7a7P|F!dEH%njy#N19seGrXWKVTAdfS>siR*^3tiFgAvH z=?gjRJa?`harw5JI!p()N|f$1Rhc3iY@>;O(P_A8g9~Jc1mXX=g8Zq=5a;>7YX)!s ze_Zze|AlrFTvLLY(>hc(5&|8!2Q`}s9tTgr`2;w>VAE^WnkoQ&OcC!EmQ#9MhAb@N zC@Sbm5UCJU2qB3=q1sagNs;U<#LX75;uK!+*T)kONbeLXn}vnCleyZLPa*;TDwIhf z<}!;VEah)LqXbi`xnZj{gDRkLvN`mFQ8zRhV6*Bni*IB1dX9LMto12m1DbW$ zF98zkAT&+XH*TploW@PwaNv*y6jNlG%vkaZL$bG*K&6!rp1O7hw~5fC=Bq&V2SxQ1 zagQJlUbGHQ;Sih3{*snN;t$>5@Hs%;x8J6?CCxNw+MXO#wB@YupxyP0fga3Vkvzma z90{B;pbckR;G7X1)Vh?_kTd9ZS{C?7xS<|R5q(f){(SRS-mJn9BZrdBD>e%aRPtLd zq*}u+`wf?kNN%;559oC+JkJr7Rth_e#yr$P?`yJr(asYS6bZqFFAn50RG+8HpMnD| z8Bf2bUafaZe!OZ%Cq?KaqF`qOd!ud&KFBZE3j~$XkY^ojh|D^we;*&q;{+N0nu`8+ zEkJO{OrH}54jv6S>j$cMMvw7R(4rAo8*rZZ0Tj~*tbm&z)|6oLBGO+YgKD_q^ak;+ zj7r9X$Dw+G8LI1c0>%Qn60*B(UzAHR(~_2G>+tSxVN5q9G;W`hk+q4s4d`IEKnB31 z7EYhSj<%e6FyX;qR!@K3yO#t11FFQxIS!Of3&= z0FyxV60OIjB)ELg1f~c?-RA}*NMKBO)4~!sNuglBKvSi=93^;&MZ_hJimJwz&r)yQ z)Kg%R?dA*cY1cv}^XkjL?N)3yoI{Hh?%Vufbc~#K-@Pq8DqtU?V@#FiuF;_fgX+!V z^g79%n_hK&4A?s6;>CZm8Y4@P4_oM3R%;0LBE{+5}L3Zd`Ur$%wAF^MfZb}*aM zLN+m%R8IkRCW%+(IGeRF^7Q7HbOQKm_#8Vh8L&%udSr_>uzuKCT1tkBozWDzqJg8N zk!V05;u`xK8FYEp-;0zz)l)mwExe@*^MY&H&+Fx30Jl-@_%|83z}c+AosVTN^T zT@oq;cmLOS7FC$lDEJxLz!qnv^AK_|DNVk*)u3j4Rul*PFCTQI0y>HiK0&Ub;TDB% z!2v5#cjE)&bS1bFe8dMfJPon<@ynR3Rm=~Ux+`E(0vF~i4w z%AeFxuin0(5*2^)=hBTs_x=4ndxj|;=jYEMA@LgzdIXw#!)bhDe4KAb{j#>Ng+F=R zjd<}@X(Stbd_O^HR{rW>A~fw52eHUsT&r$006HmbA6d67{D7*K!5{Ha6-oJQ-C+OP z8(pJZ4-qoJ#2Y7_bz@4=7~huWRlvDH%D=^j#r2w-s&YlJCp?V5QQ*s_Q|o&>RK;xq z1JALS?c*xcdEL@Yi19iD?rfL^d2*&z{{4g*H*h|3oWs}jaA$EV`D%>0%BA2RHZEb{ zZxqv`!_LmYq@fA~$UdljFvnB(TaZ1G+#&lpI`To$!O`d8sG0%}g1R*#xfWr}>%~to zho2dKP(xW(F0}q}uzl79^w{h1^A#kJ@A1Vx^HO;B-lbiunnIqKHRwChZxQl|F5kP8 zFY305mS8(;if3(zxfKG@<}1P?z(M#r7*gJPVKS5Tls8E;@Z_g8p#q&`Sh(?Y-OH;@ zKMxbz6Nbi1>*)rg?;Bo_z+WQ!qO}^~GfTb{x0qy7ZgxT%%T1(^s;$$?A2a!5o!y&4 z&<;D=hvV&KlbfFO1WiZs059l_=TX*_!F<3yS!dT}c-(qI_Hv|%;Jsx^>^&ID(Xj*w0u>||+&zGJX+a4x&sV@fCIn#8G8w2hMmJc4`&=nLSVq$yH8x5Cb6k_{~WlJ4)O+F*A}%323)1qhFjhI9*iH_VBj^3rXixQGJ?v-&BqXD1r8js zL3XAs^tJV)B5RX->>Ev&x9U@Q?d$s8NuueZn@@Ch&JcZXM)#WBvrgC6(nQ*WAAOI$ zU1JHo{#aZ0D*mCM0u^cV178u`~O9nsJ%5=3Z~FH(c4XYjxV%=K~z;yRn6JQp19w!>FB zd#*3P?N+O^kyjLBQGffbLDq>k_MyR zt<0;Hn3QVE2iwX+pVyoum_pZrXNX^~QWuS-Pl~Zqh23gAv9rNa636H)y#I~7jd@Gf zs28`~e#JlS-!W+8$Ou}x4A1N=cIdpmQKYE-Ve;c?r#}Ws;{ou?O=tM4+uXJq;yoVw zUV?P@m2-mZA7o^07Vh@s(11I>*EYI9HZ-dJ^G;1A3}&tUHO^JX>j4rhhiRkhsl~`> zQE^gER(u#_Bd(~5Jy;!o|BkNI0V{U8(n{8Kd>(d$BJmgq^_!kZSk+?lpmX=X6q@$! zKw%Ymd=zZ(`E{{4{ue#sBeIK4?MjN$J=|W0Bg>0#phvbZRhKK+zK^|5WdVl)><-hf z8)p*L)m95z10wHwsH+$KJBY#0#s>uHM?VV5JTJ3ZO$LLTmlR)OE61tOD}@*@yH_Gm z-}GWJnlB)K@2ExpSS2&NTVll9MAEQ@$RHjjE@aOUmOmD(Unz0=g~!{q^JdYocX~g= zf_dhu+rsnt-Fw#HWpz~L(^jOM`Qy;rs4@-FZ<6C~r4v`9O!i7PDwVEbg{bpx-;L`% zN#7H(J@OCu-se7UG0p1gM_K-{W@>c|Nl9j3hdeO_cMixS0C)vW`)D$1zo8f-+#QKICafJ9MpmZ&0Gat0M7=bW=5q9UL~0TmRX zK{7NsDoG?tlY`_W&>(TD-~ao*xtN<-Yu2o}nE$zOEt~EqoO7ybSJip;K3^x}xAYR# z?MRGKEsANKRj}3}wD6xd|2kbw+UPfTAW<4xc9@1?3I5Z6he?toH0Vh9gH{$Wp+r6V z!f>BH{dw%Bc|zPkAMsq3aq2tqI)aiiTAzJ=C9#@{J)6lfBIiZ`*r&dGvvBkTX zxnCZ_ZbV)IxE`8|oE%O(aAUFexo~GJ*jlVaW|cRuAc~{eYmbemz1N7=Pc{RCN}u4L z?e*Ade-4@$M`0A%Oe#FCP!mO?chh}>@FSb!8qa61PXwLX93KCh>rBeunW5^>n+X!m zI&0;V_?xiCdSB0x&*k)YUY8Jm%X7Y8+Y?me>;jKu9`Lea*UsX3Fr#DRSky6VC|Rt} zS2abm`-YmgD0Qj|RHa>WiAM1@yhWn6JcRcQg%l`hpQa$?ls7E1FFJ{(F6rN$XW!M? zKMY}<4je8L7hLWLq`>qP(tB(nW%0Njov*D6^b(*p$r78dyR~PA4PR(|Rll8H^S<>| z`o8K=!*&UJzCrKt?Ul8}TEndHqJ~fLYQw%#dXw(&XPNRt-pij{UVnZt>P&HdzVLJ& z0YR#-Ej#3xzW9!Mxa!6_>Fe(g?=T6IxHkO7@5%JPRxTjrG=>^nJH|Uao?R1lo&B=1 zvOR21A$9d4RR@=Et+|qtQpuiT;%#(J@P*jca4j(>oSzj_pJ2L9jwdS+VM|$l1lUCf z<9GovXOp`C0+ka zwVgeBb`c>FSWEz`*vw-0td>+F(=zTmRi|}ptgDsD(N*E{_IBl2OQCa?c;q|W$bl&F zX-jTzlc#Y3Q5;Ri#-r$paP8;=k56}Npb2!iS?B`Qh8{c5fq);LJ2_dO&inl5HCS#A zt%;2%z&6Ar#5CVht&HFoDv zKW$;>#^pI1z$U|=gMIdl`KXB|e60J%YlXyW>J65i(^BLJ^I`&79GU5;qS1m6e4HHr zwz=CdqO;Z5`Q`n#918Tawt|Q}S1>FRbN`y+;sV~f39y+l?x`tuX{)yHyQ{NdE?4{s zu(VLBa+YN(;<{K=J-E2jq#}O%&l>qgM`B<_LTX{ShixOD!~-NNMQw8Q=D+UAcw~Jv zon>J6yysI+*lQPhTHkFe>b>uG+}giAc@XPfTtF=#Rd4i4Lt9^O%{x?){CoRb>D_xU zsCnh6%cee??pKZ~F)i&!`|8EFt-?a{r}gNIQ{;(ZZ3?p1w&58*gx^hqL@)sdO8fv$~rnTqiw z3fGEIYt}otqGD2)VbGtdFgZA^>9{q}$B0F2d*5b?hC0$JA3uykmts#VE5}AJ{ z0{%OeKBea|@lCA-4v1^>s@%`e8riE~Rk^7X?9uAOl{YTBxo7WhM@g88$}&&0%DvEP zoQSZLUs(T)_Fd-r!TW_`W-E4$w*MeNX27D49pQdHBDPdoS!_Y#CI2S_b7T_UZ2x*9 zCT>}GnVe=nP40|t;Vhh>;q-Qt++Wo0?R3&Vo_1Czv(h4=XD-N|Y(gWiB(!a1jLGtW zYwWOkyG*#xyc&U#2l0i}V#VEB@tk{Ei2bp<_Cd#w-cOP?HRs9)jRr6PNPB;efqeJ3 z+1p)@BNgP}*7>y1&X~|<%0wTm0=mXC*|Z9Qx8jZrXa7d0YaUGDh9%0O<2pl*m&ii7 zuP1M|BWmX{>B!dZ^B_|cePaC676^>t0}oreW_JW-1dHQ(^YhnN=YUy z9DdfbZdRVw`&QQWi+X?Wq3~S-lI_mY9wsAxgH(T=Tj)KB{WiPc(IW8~a2cg8t&fEz z2=)KE7zGrmv$<-Gv?tkDoe#zJ$TAAcq>-!a-dX!4uP;+b_N&iG`YFwP$mJgeJk)q)kSf1eJKM!$CfvI1nspECl55A_~ zo+>ZDKlIzO6sj@5gR7ulaljrRE5OBjVYKJntS*V-TRqQXjjVVwFXd%IK7j*^d*v$m zbaXFCiR%xu%Ca~}%5KqX9PTV!N!?E$*X5zUlPyB2npF0Bio7$EctKSyrY+ox; zKt?P(d$WY$gJ-Rs-^>DV>uI8wi!1Bj%F4z*k6X4O7`-s{NA@6JVe!VQ^;jJ#sp|=4 zawCdRWmhd{B`;L*(+Jzi!FfbaqlT#YY_W-2x2u_7pLLb`)`Ua&#}5?Vp~07nxD#7_ zkAl8`@e^Q_k8v0>vj1hrZZ)8BgXK4_qyD?S2E2l^J@d8eB$}rRq1TB9^jlCbm%H4M zA3F|MP}O%2oCDREUT)r_`}6%Py5x%gr6qTMK743#70OW4zI-heLV}oCSP4Q#vyz=` z4|0TDpO+Qd9H)D&MyhuBQybA&qmlJToUm;+z`2s@j|ySHT$w@PgX%}oo|N|1q}L!) zz&AT-o?OqAE!QL@B58X2;T0m|W6#q#K{_$>&gReZmbWBKH0%$Th+*m!Stu^{MrEI+ z5bQpoX3w#(MAFZhxz!hnr0V4T=^A(mv(ky zz9tB{zJk$VrTiQE4WgXO7Wx{@3=HWLS=04fFUKMW%spz%m|BM4aS#zBv!WqKq)ttl zFP%QS5>`fJrRMn%RpT?oZ$IQf|C|f8AATI+YfwmoJev+gWVBybjEJm7B~D$9`Vgsh z(+#p5hva+woYN!adv^uVIGfSz0jp&VYa@&ZgrzH5Af;;S-qXWk3f_OgXZzTkbzrt~ zDngi2M~fVM zVjV-{G@;5v{&kK0pIeh7A5Td{aOdhqRA4s4Zgv>k_`e_S&K7a!kBGcrU4?d^&g;+p znoAVssrfVC(xtgRQD&0U+QuulYzZ41YN-+1%AX(1RWfGQtGBd!_{N5l5)yW_9!%mO z{1y{@cS5&{DdVvrV>Xl}gy`MxnwQ;+hZ{)N^$|Ig|L&6v2+7s3X z4NCR{E5eU{T}*s5kbgf7!OD8gXG{7jnY0sY28;+x`nnm9Q9(eyHjmpHsd}w!T| zoVV6K?eQd-B*@DjOR)H_CXNMe6X+J3#JrqgqkpdYwwLxfV`iMRt8uv9*+PS^ui-?< zu6I{YU%NCbva>VWS=sMxS^D{(HM)6Uty<(vFg9lGSIHq}%$R%!hRf>ex29$OdQxZb zb&R5Y=Y545UeH;b(E*lfn-YQ)XVfx?b|s0+E0DEbg+N}>Thg^%I7pP?!wc5PdActv z(WJ}~3p%all`G%-kNQjv;>BbVi&>l0e!rIRJW{c4TSFDY#XUx4ft5!R!&Xf`C~%mo}a0KV>D!M}!fDdJg4einNA>c64t-zKZlZ&@K* z2FOPK<2C@M9iNBSJuN%rXxYK?KA4agvaUS!$Yt0&bCgqrw3>Imm4N-lCvVG;{b9& z(MMS=#&)FKOXlycw6OKpCGRvrRqq7P%x9(F^H73QkclrVM_t5!TF=j{Pm=+a3;M^X zqQYUl?EPEO>=K?(es?!>wy#lQKGu*9pxcXQ6ZU&a8=>G(a_u54qHm7ZkIZu zKV>zkC67eL3SM_`ZYFIV*Ir3wT9VkOQWQ$E-m1hgOgZt`g^D1gW-sNQC^nL41VmEG zXUQbM5_b>Ak8i;kZXvyCqm6e2y&$yZdA(DPA_)BQ zy0jNHY(+Sj+HANWLph)@447fe+r<9Y<2!u}L4GWT?G$MG-*K?B3-xI)7T!un>OKJ~!m zso}Wzr>pZRl$7QSP-znh{Y#5rTYs8*waPFdi{R0NiT4&6$pMKsdC9&EL?UxpnppBf zRFdR~VV;_c2XkjP8Zj_cZ+?6+c`!W0?zBwt)ue`GZ7RgMX@6wTP^^v=D#`W^BOG#K z3|jI>VVw?f%F4Le?oD>V3?XLpAhkI)S+*e=ul`y<0pV)zKVvQ~#HCPT90=*~#}6|~7KpW|`*D^QG5 zUW(LfyhcugaCtNDx|0m^0pM8|JLlWfE!i+@ru6IgkTCYshJOB%)~i$oj!CH z1#Pj*r{VObNoCN5CP^H5cy>Bb86@X_nP1>y{SzxfqsnKyY2(&z3|6a_{&>CFXZT^B z9F|zD9E27M*r3DqJH0dE3ccB==s(`BEyaTu+)|htspI1-oh)L<PKq zH;w*zoZavtPr1im(x%3fBV2}!06V6V8WuMsNUc82s87QB0Td<#dzx-k^Zk&VJ^RW!5Ze%Ek%!U znP)1%QB4-H$T;YG!eIVi%{brx4T+5wax#??E4*QzVrS~LcW2- z?rOvySG$gDfr1cwq)PwKYc|;JAK2r+$tM3di(};fqPPEl=~W!HQgV&|-V5;m<-kXL zLozBTPuhv4?*R8Y0=PX=_q`_7RBi0yVd%lA1sIJ-;zodr61g8$A)@o<3?7Z*`| zyjho{`&rPRG2lmvfT^~QPB@UB+ngNk1W#t%7P@tdsxy(bGmet8#Hu-h#lJr{mjg?T z-`w0>y+2|axF8E9OWa7R0k)LXRQycMSnJjpc7Fbzc`+Iqnu4!imF?|$RG-tkKQ&!0 zuBf144c?;!dhyNLHNmcj-vni~B4a0A)b8FTd=?+CV{T5Gsfk}VNhU8ZkLB!zdw@m0 zy!KdEs1S+7e(TGZFFz+HNb&{<(tClQ+!D)0yFOMLmzcPDAD#h+h!!{o32xy9Ju%56-Ut zE|Unv@PE4hF58a~Fw-h$dJ@*Y<%|nTMBG(Zl2BpPqN~JH9dBAGQDk z_rTZ~0RjpXB_>4$U%n{n>(ks*SC=Lc3jX-%6CUs!h<&fR@;2bAk}|j^-%E+Q5n6{E-&O&i_(@V3P78n4O(v#?%ru zYQ>T=j#x83#lya9xGKKxBMqa%p7dRYtmsuh7GCP!o)<90U%vNoExQ)W@!j4ItCaNT zidR=(-(xq1Ce%I;-la?Ia&i_KYN^@TTeKNbEY}ckVj5OH9v(NwO%Cc0_Ld)7>_s@hN|Te5ON{slh={Oj z1~do=YEZ}RGKGbO)?(?DG&GkgOePnX5>z=oJUp~@bt6(zXeF4Sk8;c@Bf|tl-!63c zV6YOQXEI+33v))-2FAyUDohf9=lD6Jc(EQG6;wTpjg5Ku`9c05(xgb$%#7)iX6Cc7 zurQZw;=0Kf85v9qi;K4lhhf?|fvH*V)BTkpm6T>HH%U4KmS-O11vsx=YhOHB@sj## zT1AF%8`KbI7ZDkn4;*(A3)&k{gZnqCcFRGVuNJSDaRZZhV6Y~fOj7d7yLayjkjO2J zoP&eI%cP{pA3yE^GjzcNUiYb4h z!wxE74PewifBr;(hbbH;0&8e%Yhx1?r2&HSooXR?0KI(8ocHe$8X6j4T-t@h*wc7< ziM)ROI>0H*(AJidh?rPMM+XN^5ZR3QsQZ*%DTYm0m=egNH-`Gp|Lrclf;)?e!5iD~ z0Y-9mc-FOlC?lW=2f2zc3#4!0CfGb8?B`EyVUddD*RLa}xHX%c#PYy;IMrUPCyV(c zZ`VmlO8&MTcn^Hu)}1eyd4z;w2kfJeNF-Vek};SeZ60diz~9&xquR%ZK#!{&94y;# ziZKN?K0^EULTfF!%Tyj&rF z@OIunwRwg=1D7tiuG8a#_$1_`n^zbZ8QmAUC}Cw<59F%2Ai@21#*ji8E1pgWGj4iy z;&e~U=|kR9_m3YXfd?+`s75ov*u+HgwDfeWPkJV$qSVrz%+ z3eL4_*Lbd9Z-!Nl6}m`t#>r4>2jzg}oBbX3B6Bc6@W`D%lKJZN2?rjRSs!CIVDu0( zD9#Nx3AeKo|2J>mSb;BKZfHQEn1FjdqTPnF|I!R<+u6yf?RS}7sn?1wxGG$?5Z2t> zTy1^*=!l35X724YEYnrsk7?sC@h1P+xV642Cv+(^^gKQwSW}Y>gaG)@s(^yr(UA{4 zhmMKKGw?E4#R|dQ$9n;=&v=y&n9mV?{jeTefn66lng;_M9v)6F8&H0jHaBMzbTH{F zc#M+~E+#L}dg;<7qY|rd^ug4un-zHFxdUavR3-SSlUS?0NG`--+W3#15re>>4&jWS zB~|xx)NdzMZ*NgBK6iLZe&cT+TDM{NqhWR3=1RiOQqMKGHKl-nTD0SM2$}rx?|u}S zxP6R6WPvEo-}L^3knLly5zz@f+r#>iWAmZ zf{zdu7KRcY*a{5p_Sn3Mg#{Uu@(&OFsw_-%o1e)OD&w|6proLsjR9hPi034*A-<-j z1~hXNR8%hYWlD3Oxj&DOR{}54aCYSQz|fFWRCL%@KYy`1b$+bW7V9a^%*^t+6j(tS z1^o0|XpF zu5N*;FDxz9p;QG^ZHEeUux$Jf9=!1@wY~Yy?P%`_R&D|O5>^)ibXGbsOrW0w(Zs0S z9;=suy_}iZbGRJL8XG4kuAw<>Xza(zx3o()!+&TlQ&F*}cION`l=^O21f8EASXiQm zhN3d1eOnwVY(Q@XB9#gRJcM1Ctlu^p%+F4*`1DK%qbs%0FZi(ztgx0AMPzoiTWa7! z*DFD~&dEtT*TUjrJOonKp82|v5S`FS((Bi5i^+!1>X8s9u!9Kd3?P_b_NM_NZIjgv ztFD*qZ^d1+BxWtD2Xu^#mT8v(CF$zw-mqyS0B7>F`F%|d)(hH@{{>+QbHN|cQ><|G zWJ=S4q6%Gqk@nRlhYFv6WA#s;J;TNjN=8OLmdKx`7?3u3pPPFD#>gVJZ)HCKJ3yel z(@JA#h67?55MCh!Ei5dsK@0)+X3?r(2M!F;C#hgkM4PXfsR*tUQB68nhTiz;P=T1K zP$PE>359WJFPZy(FdZs{@#qGu1r87Vv~9=pSL@N;-Q7x{Y;tHE4QfYiag=W*+;cmp z7ArTlF_Q+o$pj=M%ATHOqdw}B1sn{|s}X_&2tc4%jhY?B`*{OaKi*uQbNdGJBIe(G z_$^RIs2eHWiy%Y_YSJ*7~me$u=$^z!);-XRrn# z0^LJ}fdB?T!%k3PFt^AXu)hIP3|FsStu}k^Ai{7zOU4+r3kpt8kj^g7&h4+SslR;r z5&>!>;M#TaRM`atNW(nu=o(g3R&ro9mTuT~kU&g|r?UVB4~X*au}Qs&Op+I1!4gt& zS%I?v4+}b7k?yF?aWhDg2Xy77<<9+q0vz~@FFzm-&6o4 zBSGOHbtWHnu>t6HEUv*rxUCLdMO$IhDt*rH( zzK`09W;`F!=#E6zt(}%&utHGVDf#U*$NI4OCA*Rvh&Rn+(5?sLfrFFQ0bEIUGbVz; zynOuW6UAHFmG$+LN1eeoWyxfz1TScCB`q)rZP3us#e3Sw>|Z_tH}{nUPeLixEekYR z7Tg)(jm^Gm*ZKIUSVtiSU8>Lj)u*DOg1sKYUD?w9DVqwDQiyV37LeXmlfD%+=Bb9uY{@cJuGQgP_^SjV^K?ZM8@E1A%+GBph7LE+D!olG*H;fa8GuY2SvEwk6sXT^xCDJ#!_-X1nHx_Rps7C%5rShe*!pFIImsykX*YJeoM zQb>`_VG|Q($ndf{qnM1YTX@{LI4$V*mu`lorE~vu*&)KpwI)=$2r$9oGCHJ)SjfmP^vxS0I;zyA6L*iT>uISP-UV2nvo2gsS# zd5fDLl0?mMPeD!*kE8f9E$ta3l92EGmx7=OY{vqeG z#Q?I!%cB7<_V)I=ED1vXu(GlOfs%qa4bjIQ0x~>UftF6hpgAa4QR0^CepYsKD=~U- z$qf?+`+t$yr9=Oy|Gf0BNFAYqB?Isl3JCK};kcj?Mh&hxl7`P94qZ-a%N#5ey(C37 zHI4;SAo^pZ-DXN~a1|UUTmzQr?I%w@tGiuppljKpb8S43au8n`lSm# z0CdWF=2gOmb|>8-p9TNU2ALZ%Ex%hGRWUfqmHcdfLebv_A zW8UO@SdFvj=x81u9*W38Fo+p92?{2r^_#Bw+?d~bFe6y`Cx8=&hlf@m?l#FwG+^NC zqRK-}#1E2*Fv-}Oz@tak92_07-Zd?a0W0+eVvK^jd+?OF5Y4NrB|ze*Q-D5O(zD}{ zWOyYwhnE={{It)&a8x}#Z$Se1GCe&W!o_|Ug8@KQY#9PxFofztEG4IE^(lX`aTy+7 zWDZ(ZQg#4$;jV)NuV$uHG+Ydl`e)H>QEW?Y5>o5NE`gq(_@X!{7Z(?8{LhN9F68dr zr=~*A=&jghtL*^KCeu0%E-LU|?+5Hr7{p~#1ISp0@}zcSYYXEgeSF`t%t+zX=G7rL zXlQ9KARhN+tnAD+52gvxIYOj*E9v<-7EQ2Je2dOwgHY)LX#e#*_N@BB!F}yoPDzP& z#^48So;FFXjllE(hwP06kbqfpdH|6ZV%z&KUx=U# zrUpZg*_uv->4`rxi<-MN-LvxrxRW40wOi{<{h;v0 z)CZ)wZ9>HX72+}_7g@Qs@0Fj+I>Y}+=<6kr8M{z#Aw=7?O`Q>7lTGJ%esp5hkn0f)7loN2b4g{!AHI7BzJd%oI8MiEmov-TfFa zFn*JA0?6rT;}>&j5wrxez36^*h`bNUGkYIX#k-Vqf0e}d+Bi_NBXHEh*=!w&%4wW> zmQr3>`OJB9m7~<4P(MRx=?Ure>n;7}$D>C_N6)OgOo5i2x2-?#bo;TvAm`r~%}-$% z1ly!|WBHs0f6rZp6ojQvBvaB85AqSB=oaS*)T9e6Hj1s8Dt%K^;-v3UFvW?!X&n{@w)hKuiGvty(W8{t>h|sMF*G#kOIXSsPf|s~dTwXrp_^9&*W*8$qb!OLs zcK21b&(F`g65Beka&vB~Zwbnj1Wi}k(B%4{2nS#R^VPsHR(==Ch)@l{ojGx-+ImXu zf(Mu`kOTm?P-`M@$E;GG43d@`p@LwT-e{=$Ht}Z=uagIOOVyJbw0fL z&hOw5)Lm^JNg$vO+y?GNT8^;(PLV;p;}*7f2MBWR5(Fpg)uyZ|vc#MjAW;Q<#Zvcq z)lyUqHsksI`?pbJNd0IQ!EEkVNZ;W%f{FVha&_}?<~b=-kTs*Fi^1&m>upWiWe3N= z%|Vvex*=&zIRlB~`edCU%HFlkif<;vcBt~r{?Y!L`^lj@z=@iS$l6*dCK*49urhoF z0#Qg&p?C)|^bnkNI|~osl9Q7ltOT4x#rXVAD5UpuErZVSjLZ80>*y+Sux^OXE-Hjl zQZBAh@#_%PL5bHuCL6LQ(5Y00GE(fddK^19enG*Ga9sR0Mp1aRJC>Z4HueJ1T7HE$ zBpy2lYR4}>K+FJ9Q7o?D;p2-mD2#^UEsP2jf=fUE%^ha~3eZ(7`5?X~Oc9K|4c}3R z3Jy>>BME@ot54ttoE?k%u($@Q(-1?UiVbC!u}Wt~%`6$hCds|>U?CH0Yxahdzv>V# zu%!$53VcytUthv)2GQ{E4Jz9p)?_9@GSf6VYIqXNe_{r!7RHbcYKu^FekVdTufR9LUho-P)8Lce}4!+f=g%e`hjj7DY!S7TjBe@53JrpG-h@V zyaaNo=-JAuNgvbH81?p!4x6LB70?*H0pfJ8va^L)5-Au|6InJYo4$gw9mrAw_Jd+V z`zV1tXGlm0sOzRw*MoZnHOf+WSU^;^leOMZTAcF-{m1sV!nSRURL;gl1{u=6n(ObV zM|7wHfV)8wr2Ei|4sKacc_o9ox;<33pcsQ4F9UwtyS;-$#BZk_IAAOrg9 zE6)hkF>2y|h6E*6@2{^<6`C$U0RZ(GESz=*#Hsfm$YZ=K^6t=hn)y>o`=7tk_|1MLKuZrp#&uaCF{tD2ns5y$fs1? z0hvid`ruuvp-qNW{)KWXR3G6Xu<#K&5ujqb`ha`{#H(TKBCxhMI;xbT%?4$<>({Td z^ygEvm#4mY^B(LRbSfcILYe^a34YeHJdiheLD`fc0Z)9w4HrfhicN9Q3xInTF6V-Z zdyx6PdqiNp+Q!CD;dA${3@APyN_Uzn+`V(B3A7loe}L4>U4!4+I12D08&Ku6TA(28 zPIwJ!VC#NQo?Iuv!yR%Eg#s+d;?C{ZC9H{4df)%E+}(JlEW(Cmm)dT|M}$dlR_Arn zj>!N-Cu5QbgXx_6QdqqDcUM_?ND>?Nuo!*|y#;XGwE}%8?D7AoOMItAbhKQ7%dw5owH$L8xU_^Wbx39z7|_t>Wn@nh zwHf9OPTY!xJsU6n)2vlat%||J=io@AVB%NO&>*0G*5IPERr{vX)NrZktFg(@h}3&+ z#H&{~4-dO2SOnMVGl6-q3YvJ7Ik{G~e-$-U=MI z7!l|Y=5{kK!TDbq8t|2P=3lz^ad$4{ece!}qlgp&adrj@9sjl0;P2|pZ9WD@=C{I1 zEADSi3{psar;9qbXM2CzGf-as6=hszvW-C(m6%LUBR_w~dDml%$ocXke0OQ1dWl8W z#wP8(fjEb7>6Lyt$EfC+quO7((hD9-YnoZ*{?4wBd=yOU6E1^X>WR$AsyT{D*{nz1 zUN65hls-@KD~KQNn0^f zf7?@{o+jlVdk6$U)?MY>58mZb$1b&Zb_?6R&^=iie0+ce0Q_XI{BZr&EnYt6?Ss^u zoaXQiw182mV(jteU>!;vcDayY_i_An=%%&1V_bQE@$cby#bcTMHJQ&G<3quym zbR8!%j{3b@Qo$_EUr_ABcKz-P8a^5FC+b>;RxKAV@zYvO%k}TNW%+lE+9H1@h)c0j z^Y#&9?`PaG^Z0my>VV>#d)^gQP6|f8*7e`!LJVsge$s`7otz)bhhF5VWg`7C#&-Ss zfsLm{LP0?po<%+zb3s6CHPz;^0E_Wem9=9mnPgxf)^wz++Sk^)e`HX(-cy&=YPzej zkK%b?ma&^HM}~^>on=me6;@NK-1e(rH4KCtc@cdJK_POAz z7hX=S6jU2_wWW~Eu1Z}tHZu8)mX=HBCNf^{%|h#m&dvfJ7)ozX`Wo;Xy{ATh3-mpR zUGerpWe#V~uiSAi+LcjL`7UkdCn2c$;yDSUgJT^nWY-0b|8cAn>l07Hf0LL67A2y$ zZ{)cE7xMtHUML#1u|}qR2P1 zDr%i*<4MV^K6~d!KuJ@}g!Oo@(qF_Bsi;bwdwF^HX=+ghylFW45ivsi`|jJ_@_>q3 z(=Gqf*1Z5Fzv^8XLw!m0^gvFMr_@AV6qGt!|Dmy=^N}tIuDj$R+4juMStEOs=?7*$ zQ4Fxx3LS+>^-p}I*dIS9)|}p9F(8ok(wPn$-#Id%@ z)YVsMK9-fY1zTUJuzQPYBm#z>r7_@t($tS5F0ZWc`b^Hj@vsCPuEr|rRek7SYuitL zYj?-Jc<`dcsPoYR)h#&L>Vr$KU)P@5X{`Dk-wQ%98Jf+KXbIInjB0KD9JHgt$xmx{ z-79B~?Z173N9m(@?G&`u2MaveCE(S-Ot*!6wwJIojy(f>dnvupEGsL zn)7cP&}^;(_Q$%`Jt?`VhP~9r!`ob*7>P;GL>7JhzND|iQrj)Ws<5(l$Fc)8apfBu zoUX&}%gzwcCH2_Rd$~Oo>5g{h6%mox9<#-RDQ)k|G_e;6%g**B$lPez<9;M`nq0HW zdu?uhkw>6ne(wo5@RA(M~>!LE6^sLEt#CK69(JB-1#UUa)NgtT)0 z8b|KtyeuyV$7S=r){%f#D`^Y3Q8^5D72e@;TwmkbD=@FS04?4n?FS|fN{SJf%T)!w z-xtZooN|IENjP$Gp346BF1-H`M?vG@YWV+r^gsLeuRj!e58~C|HMeZpGn*=AQ1F(7ZHdXP~SmV#Ze&; c|F^z)(ci8X Date: Wed, 14 Sep 2016 09:58:12 -0700 Subject: [PATCH 2/2] update w/ instructions for running in rpm-based containers, updated system containers stuff --- ...kubernetes-in-containers-on-atomic.html.md | 201 +++++++++++++++--- 1 file changed, 168 insertions(+), 33 deletions(-) diff --git a/source/blog/2016-08-18-running-kubernetes-in-containers-on-atomic.html.md b/source/blog/2016-08-18-running-kubernetes-in-containers-on-atomic.html.md index 7e26c152..d9c548f3 100644 --- a/source/blog/2016-08-18-running-kubernetes-in-containers-on-atomic.html.md +++ b/source/blog/2016-08-18-running-kubernetes-in-containers-on-atomic.html.md @@ -1,5 +1,5 @@ --- -title: Running Kubernetes in Containers on Atomic +title: Running Kubernetes and Friends in Containers on CentOS Atomic Host author: jbrooks date: 2016-08-18 16:57:25 UTC tags: atomic, kubernetes, centos, docker @@ -13,36 +13,35 @@ This "system" vs "application" division isn't set in stone, however. There's roo In particular, the key atomic host components involved in orchestrating containers across multiple hosts, such as flannel, etcd and kubernetes, could run instead in containers, making life simpler for those looking to test out newer or different versions of these components, or to swap them out for alternatives. +The [devel tree](https://wiki.centos.org/SpecialInterestGroup/Atomic/Devel) of CentOS Atomic Host, which features a trimmed-down system image that leaves out kubernetes and related system components, is a great place to experiment with alternative methods of running these components, and swapping between them. + READMORE -[Suraj Deshmukh](https://twitter.com/surajd_) wrote a post recently about [running kubernetes in containers](https://deshmukhsuraj.wordpress.com/2016/07/19/hybrid-setup-for-multi-node-kubernetes/). He wanted to test kubernetes 1.3, for which Fedora packages aren't yet available, so he turned to the upstream [kubernetes-on-docker](https://github.com/kubernetes/kubernetes.github.io/blob/master/docs/getting-started-guides/docker-multinode/master.md). -Suraj ran into trouble with flannel and etcd, so he ran those from installed rpms. Flannel can be tricky to run as a docker container, because docker's own configs must be modified to use flannel, so there's a bit of a chicken-and-egg situation. +## System Containers -One solution is [system containers for atomic](http://www.scrivano.org/2016/03/24/system-containers-for-atomic/), which can be run independently from the docker daemon. [Giuseppe Scrivano](https://twitter.com/gscrivano) has built example containers [for flannel](https://hub.docker.com/r/gscrivano/flannel/) and [for etcd](https://hub.docker.com/r/gscrivano/etcd/), and in this post, I'm describing how to use these system containers alongside a containerized kubernetes on an atomic host. +Running system components in docker containers can be tricky, because these containers aren't automatically integrated with systemd, like other system services, and because some components, such as flannel, need to modify docker configs, resulting in a bit of a chicken-and-egg situation. -### setting up flannel and etcd +One solution is [system containers for atomic](http://www.projectatomic.io/blog/2016/09/intro-to-system-containers/), which can be run independently from the docker daemon. [Giuseppe Scrivano](https://twitter.com/gscrivano) has built example containers [for flannel](https://hub.docker.com/r/gscrivano/flannel/) and [for etcd](https://hub.docker.com/r/gscrivano/etcd/), and in this post, I'll be using system containers to run flannel and etcd on my atomic hosts. You need a very recent version of the `atomic` command. I used a pair of CentOS Atomic Hosts running the ["continuous"](https://wiki.centos.org/SpecialInterestGroup/Atomic/Devel) stream. The master host needs etcd and flannel: ``` -# atomic pull gscrivano/etcd - -# atomic pull gscrivano/flannel - # atomic install --system gscrivano/etcd + +# systemctl start etcd ``` With etcd running, we can use it to configure flannel: ``` -# export MASTER_IP=YOUR-MASTER-IP +# runc exec etcd etcdctl set /atomic.io/network/config '{"Network":"172.17.0.0/16"}' -# runc exec gscrivano-etcd etcdctl set /atomic.io/network/config '{"Network":"172.17.0.0/16"}' +# atomic install --system gscrivano/flannel -# atomic install --name=flannel --set ETCD_ENDPOINTS=http://$MASTER_IP:2379 --system gscrivano/flannel +# systemctl start flannel ``` The worker node needs flannel as well: @@ -50,9 +49,9 @@ The worker node needs flannel as well: ``` # export MASTER_IP=YOUR-MASTER-IP -# atomic pull gscrivano/flannel +# atomic install --system --set FLANNELD_ETCD_ENDPOINTS=http://$MASTER_IP:2379 gscrivano/flannel -# atomic install --name=flannel --set ETCD_ENDPOINTS=http://$MASTER_IP:2379 --system gscrivano/flannel +# systemctl start flannel ``` On both the master and the worker, we need to make docker use flannel: @@ -83,9 +82,159 @@ On both hosts, some context tweaks to make SELinux happy: # chcon -R -t svirt_sandbox_file_t /var/lib/docker/ ``` -### setting up kube +## Kube in Containers + +With etcd and flannel in place, we can proceed with running kubernetes. We can run each of the kubernetes master and worker components in containers, using the rpms available in the CentOS repositories. Here I'm using containers built from [these dockerfiles](https://github.com/jasonbrooks/CentOS-Dockerfiles/tree/pr-kubernetes/kubernetes). + +### On the master + +``` +# export MASTER_IP=YOUR-MASTER-IP + +# docker run -d --net=host jasonbrooks/kubernetes-apiserver:centos --admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota --address=0.0.0.0 --insecure-bind-address=0.0.0.0 + +# docker run -d --net=host --privileged jasonbrooks/kubernetes-controller-manager:centos + +# docker run -d --net=host jasonbrooks/kubernetes-scheduler:centos +``` + +### On the worker + +``` +# export WORKER_IP=YOUR-WORKER-IP + +# atomic run --opt3="--master=http://$MASTER_IP:8080" jasonbrooks/kubernetes-proxy:centos + +# atomic run --opt1="-v /etc/kubernetes/manifests:/etc/kubernetes/manifests:ro" --opt3="--address=$WORKER_IP --config=/etc/kubernetes/manifests --hostname_override=$WORKER_IP --api_servers=http://$MASTER_IP:8080 --cluster-dns=10.254.0.10 --cluster-domain=cluster.local" jasonbrooks/kubernetes-kubelet:centos +``` + +### Get matching kubectl + +I like to test things out from my master node. We can grab the version of the kubernetes command line client, kubectl, that matches our rpms by extracting it from the CentOS rpm. + +``` +# rpm2cpio http://mirror.centos.org/centos/7/extras/x86_64/Packages/kubernetes-client-1.2.0-0.13.gitec7364b.el7.x86_64.rpm | cpio -iv --to-stdout "./usr/bin/kubectl" > /usr/local/bin/kubectl && chmod +x /usr/local/bin/kubectl +``` + +We can then use kubectl to check on the status of our node(s): -With flannel and etcd running in system containers, and with docker configured properly, we can start up kubernetes in containers. I've pulled the following docker run commands from the [docker-multinode](https://github.com/kubernetes/kube-deploy/tree/master/docker-multinode) scripts in the kubernetes project's kube-deploy repository. +``` +# kubectl get nodes + +NAME STATUS AGE +10.10.171.216 Ready 3h +``` + +### DNS Addon + +The guestbookgo sample app that I like to use for testing requires the dns addon, so I always ensure that it's installed. Here I'm following the directions from the docker-multinode page in the kube-deploy repository: + +``` +# curl -O https://raw.githubusercontent.com/kubernetes/kube-deploy/master/docker-multinode/skydns.yaml +``` + +I skipped over setting up certificates for this cluster, so in order for the dns addon to work, the kube2sky container in the dns pod needs the argument `--kube-master-url=http://$MASTER_IP:8080`. Also, I'm changing the `clusterIP` to `10.254.0.10` to match the default from the rpms. + +``` +# vi skydns.yaml + +... + +- name: kube2sky + image: gcr.io/google_containers/kube2sky-ARCH:1.15 + resources: + limits: + cpu: 100m + memory: 50Mi + args: + # command = "/kube2sky" + - --domain=cluster.local + - --kube-master-url=http://$MASTER_IP:8080 +... + +spec: + selector: + k8s-app: kube-dns + clusterIP: 10.254.0.10 + +``` + +Now to start up the dns addon: + +``` +# kubectl create namespace kube-system + +# export ARCH=amd64 + +# sed -e "s/ARCH/${ARCH}/g;" skydns.yaml | kubectl create -f - +``` + +## Test it + +It takes a few minutes for all the containers to get up and running. Once they are, you can start running kubernetes apps. I typically test with the [guestbookgo atomicapp](https://github.com/projectatomic/nulecule-library/tree/master/guestbookgo-atomicapp): + +``` +# atomic run projectatomic/guestbookgo-atomicapp +``` + +Wait a few minutes, until `kubectl get pods` tells you that your guestbook and redis pods are running, and then: + +``` +# kubectl describe service guestbook | grep NodePort +``` + +Visiting the `NodePort` returned above at either my master or worker IP (these kube scripts configure both to serve as workers) gives me this: + +![](images/guestbook-ftw.png) + +## A Newer Kube + +The kubernetes rpms in the CentOS repositories are currently at version 1.2. To try out the current, 1.3 version of kubernetes, you can swap in some newer rpms, running in Fedora or Rawhide-based containers, or you can use the latest kubernetes containers provided by the upstream project. + +### Cleaning up + +If you've already configured kubernetes using the above instructions, and want to start over with a different kubernetes version, you can blow your existing cluster away (while leaving the flannel and etcd pieces in place). + +First, remove all the docker containers running on your master and node(s). On each machine, run: + +``` +# docker rm -f $(docker ps -a -q) +``` + +Run `docker ps` to see if any containers are left over, if there are, run the command above again until they're all gone. Since we're running flannel and etcd using runc, killing all our docker containers won't affect our system containers. + +Next, you can clear out the etcd keys associated with the cluster by running this on your master: + +``` +# runc exec -t etcd etcdctl rm --recursive /registry +``` + +Then, on your worker node, clean up the `/var/lib/kubelet` directory: + +``` +# rm -rf /var/lib/kubelet/* +``` +## Containers from Rawhide + +I mentioned that the CentOS repositories contain a v1.2 kubernetes. However, Fedora's [Rawhide](https://fedoraproject.org/wiki/Releases/Rawhide) includes v1.3-based kubernetes packages. howI wanted to try out the most recent rpm-packaged kubernetes, so I rebuilt the containers I used above swapping in `fedora:rawhide` for `centos:centos7` in the `FROM:` lines of the dockerfiles. + +You can run the same series of commands listed above, with the container tag changed from `:centos` to `:rawhide`. + +For instance: + +``` +# docker run -d --net=host jasonbrooks/kubernetes-scheduler:centos --master=http://$MASTER_IP:8080 +``` + +becomes: + +``` +# docker run -d --net=host jasonbrooks/kubernetes-scheduler:rawhide --master=http://$MASTER_IP:8080 +``` + +## Containers from Upstream + +With flannel and etcd running in system containers, and with docker configured properly, we can start up kubernetes using [the containers](https://github.com/kubernetes/kubernetes/tree/master/cluster/images/hyperkube) built by the upstream kubernetes project. I've pulled the following docker run commands from the [docker-multinode](https://github.com/kubernetes/kube-deploy/tree/master/docker-multinode) scripts in the kubernetes project's kube-deploy repository. On the master: @@ -150,7 +299,7 @@ On the worker: --v=2 ``` -### get current kubectl +### Get current kubectl I usually test things out from the master node, so I'll download the newest stable kubectl binary to there: @@ -158,22 +307,8 @@ I usually test things out from the master node, so I'll download the newest stab # curl -sSL https://storage.googleapis.com/kubernetes-release/release/$(curl -sSL "https://storage.googleapis.com/kubernetes-release/release/stable.txt")/bin/linux/amd64/kubectl > /usr/local/bin/kubectl # chmod +x /usr/local/bin/kubectl -``` - -### test it -It takes a few minutes for all the containers to get up and running. Once they are, you can start running kubernetes apps. I typically test with the [guestbookgo atomicapp](https://github.com/projectatomic/nulecule-library/tree/master/guestbookgo-atomicapp): - -``` -# atomic run projectatomic/guestbookgo-atomicapp +# kubectl get nodes ``` -Wait a few minutes, until `kubectl get pods` tells you that your guestbook and redis pods are running, and then: - -``` -# kubectl describe service guestbook | grep NodePort -``` - -Visiting the `NodePort` returned above at either my master or worker IP (these kube scripts configure both to serve as workers) gives me this: - -![](images/guestbook-ftw.png) +From here, you an scroll up to the subhead "Test it" to run the guestbookgo app. It's not necessary to start up the dns addon manually with these upstream containers, because this configuration starts that addon automatically. \ No newline at end of file