Hmd<ddlZddlZddlZddlZddlZddlZddlZddlZddlZddlm Z m Z m Z m Z m Z mZmZddlmZmZddlmZddlZddlZddlmZddlmZmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&dZ'dZ(d Z)d Z*d Z+d/d Z,Gd de-Z.Gdde.Z/Gdde.Z0Gdde-Z1Gdde-Z2Gdde-Z3Gdde-Z4Gdde-Z5Gdde-Z6Gdd e-Z7Gd!d"e-Z8Gd#d$e7Z9Gd%d&e9Z:Gd'd(e9Z;Gd)d*e<Z=Gd+d,e>Z?Gd-d.e-Z@dS)0N)ENOENTENODATAEEXISTEACCESEAGAINESTALEEINTR) ConditionLock)datetime)rconf)Thread GsyncdErrorescape_space_newlineunescape_space_newlinegauxpfxescapelstat errno_wrap FreeObjectlfmatching_disk_gfidNoStimeAvailablePartialHistoryAvailablehost_brick_split)rcxtj}t|}t||z dz}||fS)Ni@B)timeint)tsecnsecs @/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/primary.py _xtime_nowr%0s8 A a&&C C7" # #D ;c:|}||j}|rw|dttjz dz}t jt d|tj||}|S)Ntimeoutz-foreign volume info found, waiting for expiry)expiry)get_sys_volinfoKFGNr rlogginginforsleep)self volinfo_sysfgn_vir*s r$_volinfo_hook_relax_foreignr37s&&((K  #F - "S%5%559 RG%''' ( ( ( 6**,, r&c i}||d<d|d<|D]W}|dkrD||}ix}|d<|r2|j|d<|j|d<|j|d<|j|d<|j|d <L||||<X|S) NopF skip_entrystatuidgidmodeatimemtimest_uidst_gidst_modest_atimest_mtime)r5eddctkstdsts r$edctrHCs CCIC     ;;AB " "C#f+ +YE YE  jF !{G !{G UCFF Jr&c tjt}tjd}|sd}tjddkrd}n|r|}ntjd}t jtd|t|| dz}t|d| zdz}tjd rt}nt}tjd rt}nt}tjd d krt}nt}Gd d|||||}|S)zDproduce the GPrimary class variant corresponding to sync modezspecial-sync-modenormalzchange-detectorxsyncz setting up change detection mode)r:MixinGPrimaryzuse-rsync-xattrsignore-deletes sync-methodtarsshceZdZdS)#gprimary_builder.._GPrimaryN__name__ __module__ __qualname__r&r$ _GPrimaryrR}s r&rX)sysmodulesrTgconfgetr-debugrgetattr capitalizeSendmarkRsyncMixinSendmarkNormalMixinPurgeNoopMixinPurgeNormalMixin TarSSHEngine RsyncEngine) excrawlthis modemixin changemixin crawlmixin sendmarkmixin purgemixin syncenginerXs r$gprimary_builderrnZss ;x D -..I   y"##w.. 3 i 122  M"7%'''(((i2244w>??IzK,B,B,D,DDwNOOJ y#$$,* +  y!""&# %  y8++!       J =z    r&ceZdZdZeZedZedZedZ edZ dZ dZ dZ d Zd Zd Zd Zd S) NormalMixinznormal geo-rep behaviorc&dt|zS)Nz%d.%d)tuplexts r$serialize_xtimezNormalMixin.serialize_xtimesr""r&cZtd|dDS)Nc34K|]}t|VdSN)r ).0xs r$ z0NormalMixin.deserialize_xtime..s(33SVV333333r&.)rrsplitrss r$deserialize_xtimezNormalMixin.deserialize_xtimes)33RXXc]]333333r&c|SrxrWrss r$ native_xtimezNormalMixin.native_xtimes r&c||kSrxrW)xt0xt1s r$ xtime_geqzNormalMixin.xtime_geqs czr&c8d|vr||d<d|vr t|d<dSdS)Ncreate default_xtimeURXTIME)r0 is_primaryoptss r$make_xtime_optszNormalMixin.make_xtime_optss9 4  'DN $ & &$+D ! ! ! ' &r&c ||jkr!|j||j}n|j||j}t |t rf|tkr[|j||j}t |t s&|jj ||j|t |t r |tkr|S|tks ||j krM|dr5t}|jj ||j|nd}||kr|d}|S)Nrrrr)primaryserverxtimeuuidstime isinstancer r secondary set_stimevolmarkr% aggregated set_xtime)r0rscpathrrt zero_zeros r$ xtime_lowzNormalMixin.xtime_lows1 $,  !!$ 22BB!!$ 22B"c"" IrW}}Z%%dDI66!"c**IN)33D$)RHHH b#   2==I ==B--H~ /\\ %//diDDDD" ??o.B r&c|j}|r;|}ttj|z|d<nt d|}||fS)Nr( )volinfocopyr rmin)r0timogapvis r$keepalive_payload_hookz"NormalMixin.keepalive_payload_hooksW\ B ,,t3ByMM b#,,CCyr&c*|Srx)r+r0s r$ volinfo_hookzNormalMixin.volinfo_hooks##%%%r&c6||krtd|zdS)Nztimestamp corruption for )rr0rxtlxtrs r$xtime_reversion_hookz NormalMixin.xtime_reversion_hooks& 999D@AA A 9r&c||kSrxrW)r0extextrds r$ need_synczNormalMixin.need_syncs Tzr&cR|jj||j|dSrx)rrrr)r0rmarks r$set_secondary_xtimezNormalMixin.set_secondary_xtimes' ''di>>>>>r&N)rTrUrV__doc__rminus_infinity staticmethodrur~rrrrrrrrrrWr&r$rprps!!N##\#44\4\\,,, * &&&BBB?????r&rpceZdZdZdZdS) PartialMixinzpa variant tuned towards operation with a primary that has partial info of the secondary (brick typically)cdSrxrWrs r$rz!PartialMixin.xtime_reversion_hook r&N)rTrUrVrrrWr&r$rrs/CC     r&rc4eZdZdZedZdZdZdS) RecoverMixinzPa variant that differs from normal in terms of ignoring non-indexed filesc8d|vrd|d<d|vr t|d<dSdS)NrFrr)rrs r$rzRecoverMixin.make_xtime_optss9 4  "DN $ & &$+D ! ! ! ' &r&c d|fSrxrW)r0rrs r$rz#RecoverMixin.keepalive_payload_hooks c{r&c t|Srx)r3rs r$rzRecoverMixin.volinfo_hooks*4000r&N)rTrUrVrrrrrrWr&r$rrsW((,,\, 11111r&rceZdZdZdS)rac|j|i|Srx)sendmarkr0akws r$sendmark_regularz$SendmarkNormalMixin.sendmark_regularst}a&2&&&r&NrTrUrVrrWr&r$raras#'''''r&raceZdZdZdS)r`cdSrxrWrs r$rz#SendmarkRsyncMixin.sendmark_regularrr&NrrWr&r$r`r`#     r&r`ceZdZdZdS)rccF|jj||dSrx)rrpurger0rnamess r$ purge_missingzPurgeNormalMixin.purge_missing s# ##D%00000r&NrTrUrVrrWr&r$rcrcs#11111r&rcceZdZdZdS)rbcdSrxrWrs r$rzPurgeNoopMixin.purge_missingrr&NrrWr&r$rbrbrr&rbc$eZdZdZdZdZdZdS)rdzlSync engine that uses tar(1) piped over ssh(1) for data transfers. Good for lots of small files. c tjtd||D]@}j|}fd}jd||d|AdS)NFilesfilesc |}|dr%tjtd|dSt |}t |t rj|dSdSNrsyncedfileT waitr-r]rrrr unlinked_gfidsaddserpbrvrFr0s r$regjobz'TarSSHEngine.a_syncdata..regjob WWYYa5 $M"XB"7"7"78884rB!"c**$+//333#t$$r&regr-r]rsyncerradd_jobFLAT_DIR_HIERARCHYr0rfrrs` r$ a_syncdatazTarSSHEngine.a_syncdatas b.../// N NA##B $ $ $ $ $ LL0%D" M M M M! N Nr&c@||jdrdSdSNTrrrs r$ syncdata_waitzTarSSHEngine.syncdata_wait/* 99T,d 3 3 4  r&cX|||dSrxrrr0rs r$syncdatazTarSSHEngine.syncdata3,  r&NrTrUrVrrrrrWr&r$rdrdsNNNN*r&rdc$eZdZdZdZdZdZdS)rez1Sync engine that uses rsync(1) for data transfersc tjtd||D]c}tjtd|j|}fd}jd||d|ddS)Nrrzcandidate for syncingrc |}|dr%tjtd|dSt |}t |t rj|dSdSrrrs r$rz&RsyncEngine.a_syncdata..regjobCrr&rrrs` r$rzRsyncEngine.a_syncdata<s b.../// N NA M"41=== > > >##B $ $ $ $ $ LL0%D" M M M M# N Nr&c@||jdrdSdSrrrs r$rzRsyncEngine.syncdata_waitRrr&cX|||dSrxrrs r$rzRsyncEngine.syncdataVrr&NrrWr&r$rere8sJ;;NNN,r&receZdZdZdZdZdZedZedZ dZ dZ d Z d Z d Zd Zd ZdZddZedZdZdZdZdZddZdS)GPrimaryCommonz(abstract class impementling primary rolerr)c|jjj|jjj}}d}|r*t |dkrt d|d}||fS)zTquery volume marks on fs root err out on multiple foreign primarys Nr)z*cannot work with multiple foreign primarysr)rrrforeign_volume_infosnative_volume_infolenr)r0fgn_visnat_vir2s r$r+zGPrimaryCommon.get_sys_volinfobs{ L  * ? ? A A L  * = = ? ?  7||a!"NOOOQZFv~r&c.|jr |jdSdS)Nrrrs r$rzGPrimaryCommon.uuidqs# < (<' ' ( (r&c.|jr |jdSdS)N volume_markr rs r$rzGPrimaryCommon.volmarkvs# < /< . . / /r&c~|jjd|j}t |t rd}|SNr|)rr entry_stimerrr r0datas r$get_entry_stimezGPrimaryCommon.get_entry_stime{s:~$00di@@ dC  D r&c~|jjd|j}t |t rd}|Sr)rrrrrr rs r$get_data_stimezGPrimaryCommon.get_data_stimes:~$**3 :: dC  D r&c|r |d}n|j}|||jk||j||fi|S)aget amended xtime as of amending, we can create missing xtime, or determine a valid value if what we get is expired (as of the volume mark expiry); way of amendig depends on @opts and on subject of query (primary or secondary). r)rrr)r0rrrrs r$rzGPrimaryCommon.xtimesW  A$CC,C SDL0$777t~c4004000r&c||_||_i|_tjddkr"t ||jjdg|_n"t ||jjddg|_d|_ d|_ tj |_ tj|_dddd|_d|_d|_d|_d|_d |_t-|_dS) NrOrPrcrawlsturnsrFr))rrjobtabr[r\SyncerrPrrsyncrrr total_turnsr now crawl_start lastreportstart change_seenr terminatesleep_intervalsetr)r0rrs r$__init__zGPrimaryCommon.__init__s " 9] # #x / / DN,AA3GGDKK!DN,@2r(KKDK   ;#<>>%&A>>  !eer&ctjdddkr,fd}t|}|dSdS)zstart the keep-alive thread zsecondary-timeoutrc dz\}}jj|t j|P)NT?)rrr keep_aliverr/)rrclsrs r$r/z2GPrimaryCommon.init_keep_alive..keep_alivesP$!88tczJJGBM(33B777JsOOO$r&targetN)r[r\rr&)r0r/r!rs` @r$init_keep_alivezGPrimaryCommon.init_keep_alivesgy,a00 !88 $ $ $ $ $ $ j)))A GGIIIII 8r&ctjr tjtjtjtjzdS#t jd}t|tr|j ttfvrYdSxYwd}t|jdztjjzdzttjjzdz}t$jt+jdd }t$j||}t/jt3d |  t%j|t$jt$jz}n#t:$rt jd}|j t<krt/jd  t%j |n;#t:$r.t jd}|j tBkrnYnwxYwt%j|t$jt$jz}nYnwxYw tj|tjtjz|t_nX#t jd}t|tr$|j ttfvr|t_YdSxYwdS) zTake management volume lock Tr)FN__subvol_z.lockmeta-volume-mntzgeo-repzlock file pathrz,Creating geo-rep directory in meta volume...)"r mgmt_lock_fdfcntllockfLOCK_EXLOCK_NBrYexc_inforIOErrorerrnorrstrrargs secondary_id subvol_numosrjoinr[r\r-r]ropenO_CREATO_RDWROSErrorrr.makedirsr)r0exfdbname mgmt_lock_dirrs r$ mgmt_lockzGPrimaryCommon.mgmt_locks     E.  0MNNNt \^^A&b'**!rxFF;K/K/K 55 DI$uz'>>K%*'(()+23 UY/@%A%A9MM w||M511 b)555666 rzBI566BB   "Bx6!! KLLLK ....*Bx6)) WT2: #9::   KEMEM9 : : :!#E   "B"g&& 287G+G+G%'"uu ts\6AABB,FAI G"!I"5HIH1I I8J AK K cbtjds/tjj|jjvStj tjds(tj dtjd|S)Nzuse-meta-volumer7z-Meta-volume is not mounted. Worker Exiting...r))r[r\r rB local_node_idrr node_uuidrErismountr-errorrYexitrPrs r$ should_crawlzGPrimaryCommon.should_crawlsy*++ O:+t|/B/L/L/N/NN Nwuy):;;<<  MI J J J HQKKK~~r&c.|dSrx)registerrs r$rYzGPrimaryCommon.registers r&FNc |r|d|_|r |df|_|}||j|_||j}t j|rdpdd|jd|jt_ |jr<|jdr.t j td|jdntd tj|jd <tj}|}|jsT|jr7t jd |jtj|jz fztj|_|j|jd z d k}|rht jd |j|jdz |j|jdz |j|j|j|jtj}t-||z t/jdkr|}|}||s|j|d|j} |jjj !dd"tG|jtj$j%g t jtd | tM t,s| kr|jj'|j(|j |) tTj+"|j,d} tTj+-| rW fdtUj.| D} | D]4} tUj/tTj+"| | 5tj0d|j1|2|rdStj0|j3|jRdSdS)Nr intermediaterz primary with volume id z ...retvalz'primary cluster's info may not be validrUzprimary volinfo unavailablerz%... crawl #%d done, took %.6f seconds<z%d crawls, %d turnsrrrzreplica-failover-intervalr|z Crawl info) cluster_stime brick_stimez .processingcg|]K}|dr4t|dddkI|LS)z CHANGELOG.r|rr) startswithr r})ryrr_s r$ z,GPrimaryCommon.crawlwrap..Qsh(?(?(?a,-LL,F,F(?,/ R0@,A,A,9!,<-=-=)*-=-=-=r&)4r3live_changelog_start_timerKNATrr,r-r]rr volume_idwarnrrrr%rWr(r&rrupdater r[r\update_worker_remote_nodestatus set_passiverrrrr stime_mntrFrArBrCrrr upd_stimerErtempdirexistslistdirremover/ set_activecrawlr))r0oneshot register_timer1 inter_primaryt0rtshould_display_infot1r`proc_dirto_purgerr_s @r$ crawlwrapzGPrimaryCommon.crawlwraps  #  " " "*.&  @.;Q-?D *'')) "49- #DI.  $7D9DDyyy" # # #) < =|H% ? R I&*l8&<>>>???;<< <"&)++ Y[[!!##.7 ,z G E#{DIKK$*,DEFGGGDJ"&*tv/F"F""L " 8 3"kDOH,EE"j4?7+CCEEE&&dk-1Z,0J'888B27||uy)DEEEE))++  * * , , ,  '')))#jjdn== $ 3 > H H3ty>>5:3J"KLL!N!N  b/<-8:::;;;"-55E"]22-77 3TY OOO}555$&7<< 0=$?$?7>>(33E(?(?(?(?2:h3G3G(?(?(?H&.EE " "',,x*C*C D D D D 1  K " " $ $ $ JJLLL  Jt* + + +o.7 ,7 ,7 ,7 ,7 ,r&ctjtdd|Dd}t |dkr|dt |dzz }|S)z;format xtime-like (sec, nsec) pair to human readable formatr|c34K|]}t|VdSrx)rA)ryns r$r{z+GPrimaryCommon.humantime..es(2I2Ia3q662I2I2I2I2I2Ir&z%Y-%m-%d %H:%M:%Sr))r fromtimestampfloatrFstrftimerrA)tpairtss r$ humantimezGPrimaryCommon.humantimebsu #E#((2I2I52I2I2I*I*I$J$J K K H( ) ) u::>> #E!H % %B r&c t|jd\}}t|}t|}d}t|jd\}}t|d\}}|dkr||d|dkrdnddz }|dkr||d|dkrd nd dz }|t j|d d t j|d d t j|d z }|S) Ngv@r^r r)yearyearsdaydaysr:)divmodrr secondsstringzfill)r0 crawl_timerrdatemshs r$_crawl_time_formatz!GPrimaryCommon._crawl_time_formatksZ_f55 tE 4yyj("--1a}}1 A:: !vv'H'HI ID 199 tqyyeef&D&DE ED fl1a0000$l1a0000&,q!2D2D2DF F r&c|j| g|j|<|j||fdfdS)z6insert @job function to job table at @path with @labelNciSrxrW)rjobrsr$z(GPrimaryCommon.add_job..sCCMbMMr&)rr\append)r0rlabelrrrs ```r$rzGPrimaryCommon.add_job~s] ;??4 ( "DK  D  %,A,A,A,A,A,A!BCCCCCr&cdtjd|z|||ddS)z7invoke .add_job with a job that does nothing just failsz salvaged: cdSNFrWrWr&r$rz,GPrimaryCommon.add_failjob..s%r&N)r-r]r)r0rrs r$ add_failjobzGPrimaryCommon.add_failjobs5 lU*+++ T5--00000r&c|j|g}d}|D]}|d}|sd}|r|d|j|g|R|S)zperform jobs registered for @path Reset jobtab entry for @path, determine success as the conjunction of success of all the jobs. In case of success, call .sendmark on @path TrFr)rpopr)r0rrBjobssucceedjrets r$rzGPrimaryCommon.waitsy{tR((  A!B%''C   '47? DM$ & & & & &r&cv|r |jj|||||dS)zupdate secondary side xtime for @path to primary side xtime also can send a setattr payload (see Server.setattr). N)rrsetattrr)r0rradcts r$rzGPrimaryCommon.sendmarksC  6 N ! ) )$ 5 5 5   t,,,,,r&)FNrx)rTrUrVrr,rfr+propertyrrrrrr+r3rPrWrYr}rrrrrrrrWr&r$rr[si22 D D   ((X(//X/  111 $$$>   000d   `,`,`,`,D\&DDD 111 $------r&rceZdZdZdS)XCrawlMetadatact||_t||_t||_t ||_t ||_dSrx)r r>r?r@rrArB)r0r>r?r@rArBs r$r+zXCrawlMetadata.__init__sG&kk &kk 7|| h h r&N)rTrUrVr+rWr&r$rrs#(((((r&rceZdZdZdZdZdZdZdZdZ dZ dZ dZ d Z d Zd Zd Zd Zd ZdZdZdZdZdZdZddZdZddZdZdZdZdZdS)GPrimaryChangelogMixinz. changelog based change detection and syncing rrr)rzM zD zE rr|rdc4dddddddddddddddd|_dS)Nr)CREATEMKNODUNLINKMKDIRRMDIRLINKSYMLINKRENAMESETATTRSETXATTRXATTROPDATAENTRY_SYNC_TIMEMETA_SYNC_TIMEDATA_START_TIME) batch_statsrs r$init_fop_batch_statsz+GPrimaryChangelogMixin.init_fop_batch_statssD    r&c^|dvrd}|j|ddz|j|<dS)N) FSETXATTRrrr))rr\)r0tys r$update_fop_batch_statsz-GPrimaryChangelogMixin.update_fop_batch_statss=   B#/33B::Q>r&cdtjtjdz} t jtj |j |d}nM#t j $r;t jtj |j |d}YnwxYw|D]} tj |}| tj |j |tj |z#tjd}t!|t"st!|t$r|jt(krY|xYw||D]} tj |}tjtj |j |Z#t"$r }|jt&jkrYd}~~d}~wwxYwdS)Nzarchive_%s.tarzchangelog-archive-formatrw)arcnamer))r todayrr[r\tarfilerGrErrFprocessed_changelogs_dir ReadErrorbasenamerrYr>rrJr?r@rcloserr)r0 changelogs archive_nametarrexcrs r$archive_and_purge_changelogsz3GPrimaryChangelogMixin.archive_and_purge_changelogss1((.*:*:*C*C I0 1 1+3+33  $,rw||D,I,8 : :"$$CC  $ $ $,rw||D,I,8 : :"$$CCC $   A G$$Q'' T%BAFF " 0 0 3 35555 lnnQ'W--W--3693F3FHIIKKK   A G$$Q'' "',,t'DaHHIIII   7el**HHHH     sE8A66AC?CA6D??AF+F+AH I(IIIctjtjdt t jj}tj d|z|S)Nz working-dirzchangelog working dir %s) rErrFr[r\rr rB local_pathr-r])r0workdirs r$setup_working_dirz(GPrimaryChangelogMixin.setup_working_dirsN',,uy77%ej&;<<>> 07:;;;r&c~d}|D]}ttj||d|}t |t sL|dz }t jtd|z||dddkrtd|j d|dS) Nrr)z %s FAILED)rr5rzEThe above directory failed to sync. Please fix it to proceed further.failures) rrErrFrr r-rUrrrk inc_value)r0r entry_key gfid_prefix log_prefix num_failuresfailurerFs r$ log_failuresz#GPrimaryChangelogMixin.log_failuress  L LGrw||KI1FGGHHBb#&& L!  bz!9&-///0001:d#w..%'KLLL j,77777r&c t}g}g}t}|D]7}|ddr|dd} n+|ddr|dd} n|dd} |dd} |dd s|ddr|dd } ttj|| } t | tr| tkrtj td |t| |dd r2| td|dd | n1| td|dd | || | dvrttj||dd} t | tr| tkrtj dt|z||dd| td|dd|ddkt | tst!| | rtj td|t| ||dd| dkr=| td|dd|dd(|dd rtjtjt$jjd| dd| dd| } tj|| dd| dd}| }tj td|t| ||kr`tj td|t|  ||df#t.$rYswxYwtd| ||| d}tj td|t| | |tj td|t| | td|dd | 8|d tkr| dvr|dd} n|dd} | dd }ttj||} t | tr`tj td!|t|  ||d#t.$rY$wxYwtj td"|t| tjtjt$jjd|dd|dd|} tj|| dd| dd}| td#||| j| j| j$9tj d%t|z|rB|D]?}|dd&vr3|d|vr)tj d't|zd(|d)<@|r|jj|}||fS)*Nr name_mismatchsecondary_entryrGrentry1entryr5 gfid_mismatchsecondary_gfidzSEntry not present on primary. Fixing gfid mismatch in secondary. Deleting the entry) retry_countrsecondary_isdirrgfidrr)rrzUnlink source %szBFixing gfid mismatch in secondary. Safe to ignore, take out entryrz .glusterfs/rz*Fixing dir name/gfid mismatch in secondaryzJFixing dir name/gfid mismatch in secondary. Safe to ignore, take out entryrrrr7linkz4Fixing dir name/gfid mismatch in secondary. RenamingzLFixing file gfid mismatch in secondary. Hardlink/Rename Case. Deleting entryr)zbFixing ENOENT error in secondary. Parent does not exist on primary. Safe to ignore, take out entryzGFixing ENOENT error in secondary. Create parent directory on secondary.rrrr:r8r9zremove_gfids: %s)rrrrz-Removed entry op from retrial list: entry: %sTr6)rr*rrErrFrr rr-r]rreprrrHrrreadlinkr rBrr}rr ValueErrorr@r>r?rr entry_ops)r0rrentriespfx fix_entry_ops failures1 remove_gfidsrpbnamer5rrFrealpath dst_entry src_entry rename_dictpargfid dir_entryrs r$fix_possible_entry_failuresz2GPrimaryChangelogMixin.fix_possible_entry_failuressii  uu N MN MGqz/* - $56E" - H- G,D!Bqz/*D Mgaj.ID M!(,!@!@!@AAA $B,,@0).&AA?0 b*J4?.27mm'='='=>>>%((F);<<<>>)00 $X*1!*V*<+21:g+>!@!@!@AAA !$5600#%;rw||/4z/D/;/=ac/B/=ac/B/= 0?0?$@$@ %'GLLhnnS6I6I"6M191D1DR1H%J%J $*  b*4AL.27mm'='='=>>>% 11$M".=9D26w-- +A+A+ABBB % 'wqz : : : :#-%%% $%+/xn5>6?b48+:+:+:K$M".F8C26{2C2C+E+E+EFFF*00====  b*O4?.27mm'='='=>>>&,, &-aj1A&B'-///000v%%##$QZ1FF$QZ0F ,,s++A.27<<W5566b#&&MM"&=1<*.w-- #9#9#9::: wqz2222%M"&E0;*.w--#9#9#9::: "{27<< 8M8D8?! 8?! 8? ,A,A B BH !# S(..2E2Eb2I-5^^C-@-@-D!F!FI!((W7)"$*")LLLMMM  (4 +=+==>>>  + + +T7DDDyL00M"QTXYZT[T["[\\\&*AlO  G-77 FFI=))s$,R  RRX:: YYcVd}d}g}g}g}g}|rd}|}|}|re||jkrZ|dz }||||\}}|s&d}tjt d|nd}|}|}|r ||jkZ|r)|D](} tjdt | %dSdSdS)NrFTr)z/Successfully fixed entry ops with gfid mismatch)rzFailed to fix entry ops %s)MAX_EF_RETRIESrr-r.rrUr) r0rrretriespending_failuresr failures2 entry_ops1 entry_ops2rs r$handle_entry_failuresz,GPrimaryChangelogMixin.handle_entry_failuressE      O#  I J" ,w1D'D'D1 *.*J*Jw +4+4'J ,',$L%,9@"B"B"BCCCC(,$ )I!+J# ,w1D'D'D  O(OOGM">W NNNN) O O$ O OOOr&c4t}g}g}t}t}|dd} d} d} d} |jdvr(|} |} | 1| /| d| dkr| } t | | dkrd} t|d} | }| n#t$rwxYw|D ]}| }||j |j }||j dd}|jd ks |jd krKtjjr:||jkr/t%jt)d ||j | r||jkr||jd vr||jkr ||j}|||jt/t0j|||j}||j}|dvrt/t0j|||j}t0j||d}t=|}||vr*t?|t r| ||dvr;t?|t s&t%j!t)d||(tEj#ds'| s%|$tK|||e|dvr/|dvrt |d}|dzrtMt |dt |dt |ddd}|$tKd||||'t0j||d6|$tK|||t |dt |dt |d|dkrAt0j||}t=|}t?|t ri}d}|r_tQj)|j*rFtWt0j,|gtZgt\t^g}t?|t rd}t/t0j|||jdz }|$tK|||||| |'t0j||ta||s&t%jt)d!||"t0j||}t=|}t?|t r&t%jt)d!||"|dkrd}|r_tQj)|j*rFtWt0j,|gtZgt\t^g}t?|t rd}|$tK|||||#|3|'t0j||H|d$krotWt0j,|gtZgt\t^g}t?|t r|$tK|||||#t%j1t)d%||&||j2kr|j3rCt0j||d|j3vrt%jd'8|'t0j||ds||j4krm|||j|dd(krtk|d)krm|'t0j||dtm|d|d|d|d)|d*+f+|'t0j||dfg|dd,vrytEj#d-d.ksatEj#d/stEj#d0r9|'t0j||dt%j1t)d1|2 t%jd3to|z|xj8tk|z c_8|j9:d4tk||j;d5xx|j8|j;d6z |j;d7z z cc<tyj<}|r:| s7|j9:d8tk||j=j>?|}tEj#d9r?|}|st%j!d=nv|tk|kr8|tk|kr%t%j!t)d>||?n+tk|}tk|}|r ||j@k|B|d@tdA|j9Cd8tk||jdvrJt | dz df}|D||j9EdB|d|j;dCxxtyj<|z z cc<| r%|xjFdz c_F|jG| |_G| |_Htyj<} |r+g}!|D]}tk|dkr |d}nt=|d}t?|t r*t%jt)dD|dEr|!$tKdF|d|G|!r|j9:dHtk|!|j=j>I|!}|B|dIdJdF|j9CdHtk|!|j;dKxxtyj<| z z cc<|j;dLdkrtyj<|j;dL<|r1|J||jKL|dSdS)MNr|rF)live_changeloghistory_changelogrTrrrrzskip ENTRY op if hot tier brick)r5)rrr)rr)rz,Ignoring rmdir. Directory present in primary)r pgfid_bnamerNr)rrr)rrr)r@r>r?rArBr)r7rrrrr)rz%Ignoring entry, purged in the interim)rr)r7rrrrz ignoring op)rtypez(ignoring data, since file purged interimrrdr=)rrFXATTROPrOrPz sync-xattrsz sync-aclszgot invalid fop type)rz entries: %srrrrrzgfid-conflict-resolutionz#Entry ops failed with gfid mismatch)countzRetry original entriesz3Successfully fixed all entry ops with gfid mismatchz$No more gfid mismatches can be fixed)rrrENTRYlast_synced_entryrzfile got purged in the interimrMETA)gor7metarrrr)Mrr*r}namerrr rG readlinesrr?strip IDX_STARTIDX_ENDr rB is_hottier TYPE_ENTRYr-r]rPOS_TYPErrrErrF POS_ENTRY1POS_GFID UNLINK_ENTRYrrrrr.r[r\rrHrrr7S_ISLNKr@rrrrr rrh TYPE_GFIDr TYPE_METArrrfiles_in_batchrkrrrrrrMAX_OE_RETRIESr r dec_valueupd_entry_stime set_fieldnum_skipped_entry_changelogsskipped_entry_changelogs_firstskipped_entry_changelogs_lastmeta_opsrdatas_in_batchri)"r0changedoneretryrclistr meta_gfiddatas change_tsignore_entry_opsr data_stimerretecrenrptrFr:rrle1entry_start_timerr num_entriesrentry_stime_to_updatemeta_start_time meta_entriess" r$process_changez%GPrimaryChangelogMixin.process_changes3iiEE LL%%b) !  9? ? ?..00K,,..J  "z'=!}{1~--( 9~~Q//#'  VS!!AKKMME GGIIII     u *u *A A4>$,./B4<==!'',,By,,,y///:(R4?-B-BM"%F(*4=(9#;#;#;<<<   B$/$9$9$- (DDDT_$$  &++Bt},=>>>,GLLb&9::<<$-(,,,0 S"T->*?@@BBB c2a511BrBU{{z"c':':{ R(((Yz"c/B/B R)28<"&N&N&NOOO  9%566J/J#NN4R+H+H+HIII777Y"2a5zz&=)&0BqE 7:2a5zz7:2a5zz9:9: &<&<&<!(tF2480:0:0:!;!;!; !& "',,sBqE*B*B C C C (NN4R(+BqE '*2a5zzs2a5zz$C$C$CDDDD8^^c400BrB!"c** B&dl2:66&' bTF8)/99%b#..&!%B/ S"T_q-@*ABBDDBNN4R-/b$:$:$:;;; IIbgll3556666.dB77! b*068t'E'E'EFFF c400BrB!"c**! b*068t'E'E'EFFF V||!*$,rz":":*!+BK"x-3UO"="=B)"c22*%)tBRr,.(0(0(0111 :!IIbgll3&=&=>>>y' bTF8)/99%b#..%$ "BTKKKMMMM R -1-/&1&1&12222t~%%&87<<RU++t/BBBM"LMMMMIIbgll31667777t~%%++Bt},=>>>a5I%%2ww!||" rw||CA'?'?'5RU=?U>@e?A!u?A!u (F(F(F'GHHHH" rw||CA'?'?&BCCCCUAAA!9]33x?? -00@49Ik4J4J@ "',,sBqE":":;;; R 6%')))**** md7mm3444 s5zz) fc%jj111    D$7  Z (%)  Y '%( (    9;; - @+- @ K ! !'3w<< 8 8 8~,66w??Hy344 1!'ll "8}} 9L$I*6"8"8"8999154+>#>#>QJE..xAAAL$>wGGH# &5666 %G 44%X66 R(N0;1=&?&?&?@@@"%g,,K#&x==L-154+>#>#>0   h 7 C C C K ! !'3w<< 8 8 8yCCC),Y!);Q(?%$$%:;;; %%&9&;A&>@@@ *+++ty{{=M/MM+++  ;  - - 2 - -2:6?31:D .)++  AL E Er77Q;;ABBr!uBb#&&M"%E*,Q%#1#1#1222##DBqE$C$C$CDDDD A %%fc,.?.?@@@>099,GG!!(D"f=== %%fc,.?.?@@@ )***dikkO.KK***  - .! 3 326)++D . /  . OOE " " "   & &u - - - - - . .s 58C.. C:c2 d}d}t|_d|_t|_|jd|_d|_d|_tj |_ |  |r|tj ddz kr|j|jD]%}||jvr|j|&|jr||jnQ|D]N}t#jt'd||||||s|xjdz c_O|rt|_|ryt/|dd dz df}||t5t7|j||||jd |jd|_t|_nd}|dz }|tj dkrt#j t'd t5t7tBj"j#| |jd |jd|_t|_|ryt/|dd dz df}||t5t7|j|||nt#j$t'd t5t7tBj"j#| |jd |jd|_| tj%du|j;|j4t#j&t'd|j|j|j|rt#j&t'd|j'd|j'd|j'd|j'd|j'd|j'd|j'd|j'dd|j'dz t#j&t'd|j'dd|j'dz|j'd |j'd!|j'd"dtj |j'd#z z$t#j&t'd%|j(dtj |j z z|ddd |d dd tS||*|+&dSdS)'NrFTzmax-rsync-retriesr)zprocessing change changelogr|rrz;changelogs could not be processed completely - moving on...rz$incomplete sync, retrying changelogsr.z$Skipping already processed entry ops)from_changelog to_changelognum_changelogszEntry Time Takenrrrrrrrr%.4fr) UNLRMDCREMKNMKDRENLINSYMdurationzData/Metadata Time Takenrrrrrr)SETA meta_durationSETXXATTr data_durationzBatch Completed)r:rYchangelog_start changelog_endrOrr),r*rr+r4rdisable_errorlogr1r2r0rbatch_start_timerr[r\enable_errorlogrrrr-r]rrIrrr r}rnlistmapchangelog_done_funcrrkr-rUrErrrhr/r.rrrrr)r0changesr6triesr7 unlinked_gfidr5rs r$processzGPrimaryChangelogMixin.processs+!ee!ee $$&&&.2+-1*,-) $  !!###a  (UY':;;a?@@K//111&*%8BBM$(;;;+22=AAA&9OOD$7888%((FM"%8/5#7#7#7888''e<<< ( a ,!!## &)ee#?v||C0045591=CNN3'''T5w??@@@55g>>> %%fd.ABBB&'#&)ee#E QJE "56666 b"='+C0@',J,J'K'KMMMNNN  %%fd.ABBB&'#&)ee#?v||C0045591=CNN3'''T5w??@@@55g>>> LB"&s27+;W'E'E"F"FHHH I I I K ! !&$*= > > >"#D   % % ' ' ' JsOOOCa H  . :  - 9 LB+/+N)-)K+/+LNNN O O O  8 L%'1'0'1'0'0'1'/' 2"T%56G%HH J J J K K K L-(3!'$*:;K*L!L(4(3(0!'y{{T%56G%HH"J KKK L L L L$ "dikkD4I&IJ#*1:#3#3C#8#8#.sKGGG1 #AGGCLL$4 5 5 1 E E E E Er&!skipping already processed changerK)rkset_worker_crawl_statusrlibgfchangelogscanr getchangesr-r.rr]rErrrfrrrr)r0rg processedprr=s @r$rtzGPrimaryChangelogMixin.crawlsl ++,=>>>((**  q  +--  = = R 2&0222333GGGGGGG #''BM>%'W%5%5b%9%9;;;<<<,,R000NN2&&&&11)<<< %%g.....r&ctjd|_tj|_||_tj |jd|_ d|_ ||_ dS)Nzchange-intervalz .processedr)r[r\r)rr6rfrrorErrFrrrkr0rvrks r$rYzGPrimaryChangelogMixin.registersf#i(9::#1#6 --// (* T\5A)C)C%$  r&)r)rx) rTrUrVrr r!r'r&r$r%r*r)r#rr,rCHANGELOG_CONN_RETRIESrrrrrrr rIrjr.rnrjrrtrYrWr&r$rrsa88IGLHHJIIJNN   &??? '''R 8 8 8a*a*a*FOOO<{.{.{.z Y8Y8Y8Y8v111 @@@@ 777   .///4r&rceZdZdZdZdS)GPrimaryChangeloghistoryMixinc||_||_tj|_d|_||_tj |jd|_ d|_ ||_ dS)Nrz.history/.processedr)changelog_register_timehistory_crawl_start_timer history_donerf history_turnsrrorErrFrrrkrs r$rYz&GPrimaryChangeloghistoryMixin.register sk'4$(5%#1#> --// (* T\5J)L)L%'  r&c  |xjdz c_|jd| t t j}t jdr#tj td tj td|j ||  r tkrttjt"jjd}t)j| d |t jd \}}t)jd kr|xjdz c_t)j}|r rtj td  fd |D}|D]l}tjtd tj|||||m||t)jd kt t j|jz }tj td||| ||jkr|jdkrfd} |t@kr t@|z } t j!| t t j|_|"dStGtI|dS)Nr)z History CrawlrNz#ignore-deletes config option is setrzstarting history crawl)rretimerz.glusterfs/changelogsr sync-jobsrcxg|]6}t|dddk4|7Srrrs r$rcz7GPrimaryChangeloghistoryMixin.crawl..FsK!K!K!Kq$' R(8$9$9JqM$I$I"#$I$I$Ir&rrKzfinished history crawl)endtimerrr)%rrkrrr rr[r\r-r.rrrrrErrFr rBrrr history_scanrhistory_getchangesr]rrfrrrrrCHANGELOG_ROLLOVER_TIMEr/rtrrA) r0end_timechangelog_pathr actual_endrgrrhistory_turn_time sleep_timer=s @r$rtz#GPrimaryChangeloghistoryMixin.crawls} a ++O<<<((** ty{{## 9% & & , LA)+++ , , ,  R0"0(&$($8$8$:$: <<< = = =  %Z722"$$ $ ej&;&=??(:  qM  Ik " " $$Z)++a// KK1 KK$799G + +L$6*4"6"6"6777!K!K!K!KG!K!K!KI'++ b)L3573C3CB3G3G'I'I'IJJJ00444r****  ) )' 2 2 2!)++a//$  ,,t/LL R0 *"1133$($8$8$:$:<<< = = = 4 4 4!A%% $'>>>!8;L!LJ :&&&03DIKK0@0@- .c*oo>>> 5 4r&N)rTrUrVrYrtrWr&r$rr s7   M?M?M?M?M?r&rcneZdZdZdZddZdZgfdZdZdZ d Z d Z d Z dd Z gd fdZdZddZdS)GPrimaryXsyncMixina> This crawl needs to be xtime based (as of now it's not. this is because we generate CHANGELOG file during each crawl which is then processed by process_change()). For now it's used as a one-shot initial sync mechanism and only syncs directories, regular files, hardlinks and symlinks. i Nc ||_d|_g|_g|_d|_||_tjtd|jtj |jd|_|j|_ d|_ tj|jn_#t $rRt#jd}|jt(kr%tj |jrnYnwxYwtj|jD]N}|dr7tjtj |j|OdS)Nrr^z Working dirr8rKr)zXSYNC-CHANGELOG)rkcountercomliststimesr)rror-r.rrErrFrrrKrJrYr>r@risdirrqrbrr)r0rvrkrLrs r$rYzGPrimaryXsyncMixin.registervsb     --//  R !\+++ , , ,w||DL':: (, %   K % % % %   "Bx6!!bgmmDL&A&A! DL)) 9 9A||-.. 9 "',,t|Q77888 9 9s#B==ADDcfd}t|}|tjt djd j d}|ddkr7tjt d dS|dd krctjt d |d   |d gd |d gn|ddkrXtj t d|d  |d d |d dn#tjt d|n$#t$rt!jd YnwxYw{)z event dispatcher thread this thread dispatches either changelog or synchronizes stime. additionally terminates itself on receiving a 'finale' event c0dSrx)Xcrawlrsr$Xsyncerz)GPrimaryXsyncMixin.crawl..Xsyncers KKMMMMMr&r1zstarting hybrid crawlrz Hybrid CrawlTrfinalezfinished hybrid crawlrKzprocessing xsync changelogr)r8rzsetting secondary time)rzunknown tuple in comlist)rN)rr&r-r.rrrkrrrrjrr]rnrh IndexErrorrr/)r0rr!items` r$rtzGPrimaryXsyncMixin.crawls-      ' " " "   R/"1133555 6 6 6 ++N;;;  |''**7h&&L$;*.*=*=*?*?"A"A"ABBBE!W''L$@)-a"2"2"2333LL$q'A...55tAwi@@@@!W''M"%=*.q'#3#3#3444NN471:tAwqz::::L$>*."0"0"0111    1  ' s>>>:;;;;;r&c   tj|jdt t t jz|_t|jd|_ dS#t$rwxYw)NzXSYNC-CHANGELOG.r) rErrFrorAr r xsync_changerGrr?rs r$rGzGPrimaryXsyncMixin.opensz  "  03s49;;7G7G3H3HH!J!JD 4,c22DGGG     s A2A66 Bct|ddrf|jtj|j|jd|_dSdS)Nr)r^rflushrEfsyncfilenorrs r$rzGPrimaryXsyncMixin.closese 4t $ $  GMMOOO HTW^^%% & & & GMMOOODGGG   r&c|jSrx)rrs r$fnamezGPrimaryXsyncMixin.fnames   r&c>|j||fdSrx)rr)r0rrs r$putzGPrimaryXsyncMixin.puts" T4L)))))r&c||jdkr(|d|d|_|st jddSdS)z"schedule a processing of changelogrrKr)N)rrrrrr/)r0lasts r$ sync_xsynczGPrimaryXsyncMixin.sync_xsyncsb r?rArBrrerrst_nlink)r0rxtr_rootrrrdemrrrNrrFrrstime_to_updatenlinks r$rzGPrimaryXsyncMixin.Xcrawls; 3;; KK1 KK /zz#t~66H(C(( /v%%L%5*2"4"4"4555 .jj c3   > L< = = =jjt~.. c3   &f}} R!0%)&)+++,,,%C#x   y %C~~dC-- s{{s }d333 F !!$S111 kD()))l!))$//,%**400 gs # # ( L0!%''' ( ( (J 5J 5AE T1%%A**Q--C#s##  R 1%&&+oc&:<<<===>>!S#.. $**1--B"c""  R ;%&((()))~~a,,  b!;&')))***<&++A..D$$$  R 0%&((()))B LL4<#3#3=#'<#3#3=#'<#3#3=AA;< =LL|t555t{E222  |B* 5'')-wB),QQ9M-/W\\'5-I-I:K:K)LMMM''dIs29~~.1")nnc"*oo.1"+.>.>.1"+.>.>.@AAA Ax((("%1O&)$*H#&N&NO ""A#78888b!! 5''$ +? We44,6,678888b!! 5  A::++C-17CGG-0VVSVV-A135rs  HHHHHHHHHHHHHHHHHH%%%%%%%%********************************    .''''\R?R?R?R?R?&R?R?R?l     ;   11111;111*'''''&'''         11111v111      V   !!!!!6!!!H     &   FG-G-G-G-G-VG-G-G-T (((((V((([ [ [ [ [ ^[ [ [ |Y?Y?Y?Y?Y?$:Y?Y?Y?xD.D.D.D.D./D.D.D.N     9   )))))d)))XUUUUUVUUUUUr&