Social network analysis with Hadoop

Jake Hofman

Description

Social network analysis with HadoopJake HofmanYahoo! ResearchOctober 2, 2009Jake Hofman (Yahoo! Research) Social network analysis with Hadoop October 2, 2009Social networks• Rapid increase in amount and variety of social network data• Valuable information for products (recommendations, advertising,etc.) and research (structure/dynamics, diffusion, etc.)Jake Hofman (Yahoo! Research) Social network analysis with Hadoop October 2, 2009Social networksGoal: to enable analysis of large-scale social network data with readilyavailable software/hardwareJake Hofman (Yahoo! Research) Social network analysis with Hadoop October 2, 200911970s∼ 10 nodes456 JOURNAL OF ANTHROPOLOGICAL RESEARCH FIGURE 1 Social Network Model of Relationships in the Karate Club 34 1 3 33 2 27 8 26 i 9 25 10 AND IN SMALL GROUPS CONFLICT FISSION 453 to bounded social of all in all the data groups types settings. Also, can be collected a reliable method familiar to required by currently the use of nominal scales. anthropologists, 19 18 16 17 THE ETHNOGRAPHIC RATIONALE This is the of the social the 34 indi- graphic representation relationships among The karate club was observed for a of three from 1970 period years, viduals in the karate club. A line is drawn between two points when the two to 1972. individuals In addition to direct interacted the in contexts of outside the club those of to being represented consistently observation, history prior karate and club Each such line ...

Informations

• Publié par: mtoledan
• Publié le: 21 novembre 2011
• Nombre de lectures: 81
• Langue: English
• Poids de l'ouvrage: 4 Mo

Extrait

Jakman(eHof!oeRaYohhcS)esraetlniaocalanrkwohtiwsisycOpoodaH009

Jake Hofman

Social network analysis with Hadoop

October 2, 2009

Yahoo! Research

oteb2r2,

mfoHekaJoohaY(naroakenwtiswsanylearc!Rescialh)So

•

Rapid increase in amount and variety of social network data

•

Valuable information for products (recommendations, advertising, etc.) and research (structure/dynamics, diffusion, etc.)

202,

Social networks

09tcOpreboHhtiooda

(YanfmHoes!RooahoS)hcraewtenlaicaJek

networks

09

to

enable

Social

Goal:

with

readily

analysis of large-scale social network available software/hardware

data

orkanalysiswithHdaooOptcbore,202

boreOptc90

10 1

,202siswnalyadooithHlaicoS)hakrowtenooah(Yanrceaes!R

E.g.

karate club (Zachary, 1977)

edges

1970s

1970s

∼

∼

•

Few

nodes

•

highly

detailed

direct

observations;

nodes

and

info

on

keJafmHo

nalyorkanetwcialOptcdaootiHhisws

1990s

nodes

10 4

∼

09202,erob

E.g. APS co-authorship network (htthjmyla/sp80:p//ib.t)

edges

and

nodes

on

details

few

relatively

samples;

indirect

Larger,

•

•

fmHo(YankeJacraeoS)hoohaseR!

laicoS)hcraeseR!ooah(YanfmHokeJaboreOptcdaootiHhsiswnalyorkanetw

•

•

02,2

∼10 8

90

metadata

nodes +

Present

Present

E.g. Mail, Messenger, Facebook, Twitter, etc.

Very large, dynamic samples; many details in node

and

edge

Soh)alcies!Rrceaylanwsiswtenakro machinesommodity naY(haooaJekoHmf ussprkwoetcntitacroftimilyromemhels,iSpm daootiHhboreOptc09

Scale

2,20

• Example numbers: • ∼10 7 nodes • ∼10 2 edges/node (degree) • no node/edge data • static • ∼8GB

eResrahc(naYoh!oakeHofmaJ

Simple, static networks push memory limit for commodity machines

eb2rcOot9

• Example numbers: • ∼10 7 nodes • ∼10 2 edges/node (degree) • no node/edge data • static • ∼8GB

Scale

2,00wiisysalopdoHathnlaicoS)nakrowte

estoragesiderabl uqri c e no miti;ser Soh)alcies!RrceaY(naoohaekaJmfoHoberpOctadooithHiswsanylroakenwt

• Example numbers: • ∼10 7 nodes • ∼10 2 edges/node (degree) • node/edge metadata • dynamic • ∼100GB/day

Scale

9002,2 yDanadatim,ceecxeskrlyromemdochsic-rwoetlnia

Dynamic, data-rich social networks exceed memory limits; require considerable storage

eb2rcOot9

• Example numbers: • ∼10 7 nodes • ∼10 2 edges/node (degree) • node/edge metadata • dynamic • ∼100GB/day

Scale

2,00iswialysdoopthHamaofeHakJrahceResoh!o(naYrkanetwoialn)Soc

nakasilynealortw)hcricoSR!ooaesefman(YahJakeHoadhHitswobctpOoo9002,2re

MapReduce convenient for parallelizing individual node/edge-level calculations

Distributed network analysis