1Segmenting Modeling and Matching Video Clips Containing Multiple Moving Objects

Svetlana Lazebnik

Description

Niveau: Supérieur, Doctorat, Bac+8
1Segmenting, Modeling, and Matching Video Clips Containing Multiple Moving Objects Fred Rothganger, Member, IEEE, Svetlana Lazebnik, Member, IEEE Cordelia Schmid, Senior Member, IEEE Jean Ponce, IEEE Fellow Abstract— This article presents a novel representation for dynamic scenes composed of multiple rigid objects that may undergo different motions and are observed by a moving camera. Multi–view constraints associated with groups of affine–covariant scene patches and a normalized description of their appearance are used to segment a scene into its rigid components, construct three–dimensional models of these components, and match in- stances of models recovered from different image sequences. The proposed approach has been applied to the detection and matching of moving objects in video sequences and to shot matching, i.e., the identification of shots that depict the same scene in a video clip. Index Terms— Affine-covariant patches, structure from mo- tion, motion segmentation, shot matching, video retrieval. I. INTRODUCTION THE explosion in both the richness and quantity of digitalvideo content available to the average consumer creates a need for indexing and retrieval tools to effectively manage the large volume of data and efficiently access specific frames, scenes, and/or shots. Most existing video search tools [1], [2], [3], [4] rely on the appearance and two-dimensional (2D) geometric attributes of individual frames in the sequence, and they do not take advantage of the stronger three-dimensional (3D) constraints associated with multiple frames.

• between fixed


• rigid components


• vectors joining


• projection model


• coordinate vectors


• patch parameters


• approach has


• local


• moving independently

Sujets

• Google
• Schmid

Informations

• Publié par: profil-zyak-2012
• Nombre de lectures: 7
• Langue: English
• Poids de l'ouvrage: 3 Mo

Extrait

Segmenting, Modeling, and Matching Video Clips Containing Multiple Moving Objects Fred Rothganger, Member, IEEE, Svetlana Lazebnik, Member, IEEE Cordelia Schmid, Senior Member, IEEE Jean Ponce, IEEE Fellow

1

Abstract — This article presents a novel representation for In principle, our patch-based 3D object representation can dynamic scenes composed of multiple rigid objects that may be naturally extended from modeling static objects captured undergo different motions and are observed by a moving camera. in a few photographs to modeling dynamic scenes captured in Multi–view constraints associated with groups of afne–cvoariant scene patches and a normalized description of their appearance v c i h d a e ll o en s g e i q n u g en b c e e c s a . u I s n e p it ra r c e t q ic u e i , r tho se u v g e h r , a t l hi s s ig e n x i t  e c n a si n o t n m is od q i u  it ac e - are used to segment a scene into its rigid components, construct es three–dimensional models of these components, and match in- tions of the existing approach. First, our previous work [5] has stances of models recovered from different image sequences. e The proposed approach has been applied to the detection and h as a s n u dl m e e t d he a s s i i g m n p i l  i ca e n d t a p f e  rs n pec p ti r v o e jec e t f i f o e n cts m c o o d n e t l, ain w e h d ic i h n c m an a n ny ot matching of moving objects in video sequences and to shot scenes from lms or commercial video — see, for example, matching, i.e., the identication of shots that depict the same scene in a video clip. the street scene from the movie Run Lola Run (Fig. 6). In the in Index Terms — Afne-covariant patches, structure from mo- present contribution, we address this issue by introduc g a tion, motion segmentation, shot matching, video retrieval. n p o er v s e p l e p ct r i o v j e ec e t f i f o e n cts m b o e d t e w l e f e or d s if u f r e f r a e c n e tp p a a t t c c h h e e s s , t b h u a t t u a s c e c s o a u n nt a s f fo n r e n model within each individual patch. Second, video clips almost I. I NTRODUCTION always contain multiple objects moving independently from T v H i E de e o x c p o lo nt si e o nt n a i v n a b il o a t b h l t e h t e o r t i h c e hn a e v s e s ra a g n e d c q o u n a s n u ti m t e y r o c f re d a i t g e i s ta a l i b e n a y t c o h de g o v r t e o h l e u o r p p s i a n n t g d ha f a r t o m m m e o t t h v h e o e d c t a o fo g m r e e t r s h a e e . g r W m r e i e g n a i t d i d l n d y g res a al s n l t d t h r i d a s i c s c k c o e a m d rd p f i li e n c a g a t t u t i r h o e e n s need for indexing and retrieval tools to effectively manage the  features that do not fall into any rigid group. Notice that this large volume of data and efciently access specic frames, scenes,and/orshots.Mostexistingvideosearchtools[1], i c s a f n u n n o d t a re m p e r n e t s a e ll n y t a ce r r i t g a i i d n m ki o n d d e s li o n f g v f i r d a e m o e c w o o n r t k e , n a t, n s d u t c h h er a e s fo f r a e st i - t [2], [3], [4] rely on the appearance and two-dimensional (2D)  moving people or animals. geometric attributes of individual frames in the sequence, and they do not take advantage of the stronger three-dimensional  Our approach to constructing 3D representations of video (3D)constraintsassociatedwithmultipleframes.Inthispre- i cl m i a ps ge ex s t e r q a u ct e s nc af e,  a n n e d -c t o h v e a n ri s a i n m t u p l a t t a c n h e e o s, us t l r y ac s k e s g t m h e e n m ts th th ro e u t g r h ac t k he s sentation, we propose a richer representation of video content formodelingandretrievaltasksthatisbasedonexplicitly t a h n e d s b ce ui n l e d . s T 3 h D er m es o u d l e ti ls ng of 3D eac m h od ri e g ls id re c p o re m s p e o n n t e t n h t e p s r t e r s u e c n t t ur i a n l recovering the 3D structure of a scene using structure from motion (SFM) constraints.  content of the scene, and they can be compared and matched Following our earlier work on modeling and recognition  u m s a i t n c g hi t n ec g , h i n . i e q ., ue r s ec s o i g m n il i a zi r n t g o s th h o ot s s e o in ft [ h 5 e ]. s T a h m is e i s s ce u n se e f [ u 1 l 4 f ] o , r [ s 1 h 5 o ], t of static objects from photographs [5], we propose to repre- sent 3D structure using a collection of small planar patches,  [16], [17], [18], [19] — a fundamental task in video retrieva l. combined with a description of their local appearance. This  The rest of the article is organized as follows. Section II approachuniesrecentworkonlocalimagedescriptionusing t s i u o m n m II a I riz d e e s sc r r e i l b a e t s ed ou w r or a k pp i r n oa v c i h de t o ot a r n a a c l k y i s n is g a a n f d n S e F -c M o . va S ri e a c n - t afne-covariant regions [6], [7], [8], structure from motion [9],  patches in image sequences, identifying subsets of tracks that [10], [11], and shape from texture [12], [13]. It is based onthefollowingkeyobservation:Althoughsmoothsurfaces r m ig o i v d e c r o ig m id p l o y n t e o n g ts e . th S e e r, ct a i n o d nI b V uil d d e i v n e g lo 3 p D sa mo m d e e t l h s o o d f f t o h r er m e a s t u c l h ti i n n g g are almost never planar on a global scale, they are always planarinthesmall—thatis,sufcientlysmallsurfacepatches I 3 V D -B mo pr d e e s ls en c t o e n x st p r e u r c i t m e e d n f t r a o l m re d s i u f l f ts er o e n nt s s e h v o e t r s a . l S v e i c d ti e o o n s, s i I n II c - lu F d a i n n d g can always be thought of as being comprised of coplanar  shots from the lms Run Lola Run and Groundhog Day . points. The surface of a solid can thus be represented by a collection of small planar patches, their local appearance, and  Section V concludes with a discussion of the promise and adescriptionoftheir3Dspatialrelationshipexpressedinterms l f i u m tu it r a e tio w n o s rk o . ftheproposedapproach,togetherwithplansfor of multi-view constraints.  A preliminary version of this article has appeared in [20]. Fred Rothganger is with Sandia National Laboratories, USA Svetlana Lazebnik is with the Department of Computer Science and Beckman Institute; University of Illinois at Urbana-Champaign, USA II. B ACKGROUND Cordelia Schmid is with INRIA Rh ˆone-Alpes, France Jean Ponce is with University of Illinois at Urbana-Champaign and with As stated in the Introduction, the main target application EcoleNormaleSup´erieure,Paris,France for the approach presented in this article is video indexing