[Python]큰 용량(사이즈) 데이터 셋 처리 방법과 처리 시간 비교

 

목차

 

 

 

 

 

 

 

 

[Python]리스트에 값이 많을 경우 효율적으로 처리하기

 

통계나 빅데이터, 머신러닝 등 처리하고자 하는 입력값의 크기가 큰 경우들이 있습니다. 데이터의 정렬이나 출력등 모든 값들을 불필요하게 다 꺼내서 처리하면 시간이 많이 소요 됩니다. 

 

하지만 Numpy 라이브러리를 사용해서 데이터를 처리하면 불필요한 정보는 효율적으로 생략하고 처리에 필요한 과정만 진행합니다.  일반 리스트로 값을 처리하면 파이선의 type을 확인하면 아래와 같이 클래스 list 이지만 numpy의 경우는 ndarray로 처리 됩니다.

 

<class 'list'>
<class 'numpy.ndarray'>

 

 

 

 

 

[Python]큰 용량(사이즈) 데이터 셋 처리 방법과 처리 시간 비교 예제

 

아래 예제 코드를 보시면 numpy를 사용한 데이터 처리가 왜 유용한지 확인할 수 있습니다.

 

예제 코드>>

import random
import numpy
import time

def rand_no0():
    a_list = []
    for _ in range(100000):
        a_list.append(random.uniform(1,10))
    return a_list

def rand_no1():
    a_list = []
    a_list = numpy.random.uniform(1, 10, 100000)
    return a_list


cur_time = time.time()
print(rand_no0())
print("rand_no0 time : ",float(time.time()-cur_time))

cur_time = time.time()
print(rand_no1())
print("rand_no1 time : ",float(time.time()-cur_time))

5~9번 라인: 일반 random 함수로 1~10사이 실수(float)를 100000개 만들어서 list에 넣음

11~14번 라인: numpy라이브러리를 사용해서 동일한 랜덤값을 실수로 만들어 넘피 리스트에 넣음

13번 라인: 7번 라인과는 달리 별도의 for 문 없이 제작가능

17, 21번 라인: 함수 실행 전 시간 측정

18, 22번 라인: 각 함수의 출력 값을 print

19, 23번 라인: 처리 시간 측정

 

 

결과>>

[7.791652657892609, 6.5769964691634115, 9.065389339268876, 9.155064404007923, 5.086772032150945, 1.2835880988766077, 1.4271912780518632, 3.4632976123232773, 9.132658035450374, 5.577726425003725, 3.869546995876078, 9.376265456423942, 1.9346289352791652, 4.427112359021556, 2.9633934241770445, 3.5516176785309126, 7.166811054534753, 6.955950116194504, 3.5282577812603995, 3.440582503967208, 5.112039146549375, 1.763510177874616, 5.933116084156813, 9.952424032344336, 1.797618270566693, 4.433605114068168, 4.579355690999761, 7.804212794525436, 6.589959535582324, 4.43155940531568, 7.690778006348381, 3.2795533412482287, 1.591815212927235, 7.023227015468303, 9.390015817938755, 9.409700160324618, 3.125847464656733, 5.554578639531853, 5.5701105581627814, 6.78241674201397, 5.939135902369833, 1.4251532504831321, 2.22233156185566, 7.863091496522106, 2.3116896842431385, 5.071397249036974, 7.967808792468396, 4.8271300121214455, 1.206364178471123, 4.391755848914539, 5.173769953857045, 5.886585133911287, 2.4101867230576106, 6.99573695983924, 6.1542544640279315, 6.462123241446037, 7.285404535147917, 2.9628539967736005, 2.531272955589074, 3.805247751034303, 7.096186333210286, 7.273114676407147, 6.6552060354268185, 7.595966201465023, 6.263834151037352, 7.845355982350866, 6.762146011183969, 4.115639272094716, 8.806322774288063, 1.9794952122186498, 3.064050405420547, 7.558428818657031, 7.161574817724421, 5.442692618679571, 8.143879485937875, 7.305279472820731, 3.306452812228338, 2.5202680105176047, 2.856318154905049, 4.104726914126585, 2.761849769485404, 8.85262330793126, 1.8983173666979034, 7.890114517620954, 6.127379838534665, 1.7473583752347956, 3.5135310233366193, 8.602447954139649, 5.83073543204581, 5.953889319235951, 4.189689033308873, 9.538998073570792, 8.681659298022907, 2.906164643307583, 1.881569041425481, 2.278868359223786, 4.55146937816791, 9.106741461940475, 5.062672242359179, 6.60916547714596, 1.4686230086925147, 8.030567965334047, 5.42563685435233, 1.6456983660357225, 6.492671455475765, 2.6508534209831733, 3.9573241589342554, 1.7432247351636492, 8.520609012085387, 3.751025511105034, 9.300296652548191, 8.53620825185887, 5.742887278366819, 9.686187552272122, 8.38443526211373, 9.508773641605657, 8.074671414659022, 3.8206697541256767, 4.974713826758776, 1.7664828674805304, 2.893834766424009, 1.1990903657353464, 3.51111499472794, 8.9662675763004, 3.441875468339555, 7.641980056875523, 1.824637322381423, 7.2076521009048085, 1.6976969459422047, 8.186245888886397, 9.041529556609719, 6.717931661725561, 2.2386430919271616, 3.3325676886732323, 2.4815779114562417, 9.439253736877598, 5.096653333693254, 1.8400032061695684, 7.609046113798996, 6.2414568266558685, 5.472306558897868, 9.704317976835876, 6.205975946051927, 3.986709533539803, 4.9979176915212715, 7.036989757490903, 9.14816160365166, 1.1317476550540473, 8.631435511952544, 7.669087031748997, 1.2893446100332921, 2.243020517757222, 7.943688415751248, 8.050323958426402, 5.486048859921992, 3.0625275952353266, 5.457951208573425, 3.4712884627966063, 1.040777530870014, 8.256333123958857, 1.9594138319573338, 9.84094192166003, 7.403183431783486, 4.244422320268195, 8.384178754977555, 7.083466418519479, 3.7069402329784515, 4.2904136480440584, 2.2183850514176675, 9.15614372185723, 6.3886174977818495, 3.2578913008001353, 9.655897177351479, 6.725678074177248, 7.9102414279613855, 3.5984055376393225, 5.4389151688451935, 9.989601685456227, 1.9737132621566458, 6.700667538076316, 7.253324443890272, 6.081659844215677, 6.957275042845652, 4.95778369087272, 7.971010564072095, 5.789749277211506, 8.60812040879406, 5.5358319601879815, 7.49873453405845, 2.861708665231122, 8.863717410798163, 2.0485980800557053, 6.417470491861051, 2.64314320993004, 5.7032885586991275, 2.8006098666841504, 3.37650007360416, 3.4679739720192635, 3.4184977317919993, 6.093253646830815, 4.209828475031237, 7.115841061973036, 1.3562763044489219, 1.8037759792016517, 1.834410119368291, 9.003637015017208, 5.196794123563299, 1.5652525974203657, 6.524781593001042, 5.629299974733562, 9.272673784812637, 6.943554043739298, 7.531940117448359, 7.322994998869298, 7.461755417739413, 1.2888083472010778, 9.132626249919843, 4.612159955207297, 3.647780001131515, 7.782970443934571, 2.633980401471525, 8.211726195849769, 6.254319450390362, 5.725368676874719, 5.608135589032499, 4.808559431142038, 8.848911691016863, 2.4857615380439655, 6.5762850793820435, 1.5546260033226962, 6.605765884957747, 5.797449418278523, 4.058320828446761, 1.8735011981259415, 4.389488541731392, 6.091817229669146, 8.605460149866811, 3.789837481863531, 7.996963083747843, 8.747385381393592, 8.92127451702562, 8.158889100825363, 7.537267979954492, 7.389351898185975, 2.46978560792075, 2.763010198569181, 8.98874496640358, 2.7824461419384923, 3.9000911267600475, 5.921930306663109, 3.1369186387219963, 7.3031126692160955, 7.497841207290204, 5.168305780402813, 8.500876757484708, 8.984905938292862, 8.110256906456579, 5.838404623267534, 8.693358691732683, 5.891340577175545, 7.420518579894937, 3.8009405985538236, 4.377739410953624, 6.473397360280417, 3.5359426858600562, 3.0107548409018787, 8.176816585147373, 3.3870904610805157, 1.9623933626951793, 6.228106611753992, 8.511418201366165, 
3.202004128677842, 7.437481101512212, 1.8186742351308225, 2.475076021605013, 7.240848493517416, 1.547118701939639, 1.5755518909865989, 8.881159596568152, 3.6200648144290977, 6.210741613821255, 6.94952783541999, 4.966250789912674, 9.920777821056276, 9.037993684176229, 2.412793123133283, 8.267407330187702, 8.926194911573324, 2.590206277652556, 8.737066672008016, 2.105525344417117, 6.336644837377401, 9.894190939964924, 1.7864079738701102, 5.323581220803491, 9.788221682785585, 9.003376401319759, 9.150690583121484, 3.920102185404606, 8.793295617865606, 9.458318775973323, 4.73300533276569, 1.6618677968773272, 6.8375584411310095, 4.410338445079676, 1.9478945482645469, 7.077322095346347, 9.479697258905052, 3.8466898927611264, 3.0206478638165724, 6.439083445170086, 4.491271360849019, 3.8024530900617477, 2.5269098693300993, 5.6415648512258985, 6.791663539193022, 5.196419283837749, 3.018942482325687, 7.688201756817553, 7.356844584651086, 7.851669854701145, 6.426791646929326, 5.663132937898128, 2.6214555288129686, 1.0898127066466587, 3.40216639642938, 6.376361772162653, 7.0234941809645575, 7.206986675268688, 4.294436338863003, 1.3749222025722019, 4.946156480172496, 5.161859781351751, 3.918874489891619, 7.76299607665169, 5.06556442873421, 5.123396081919869, 4.101973059123138, 5.4330140837366665, 5.0463855624173375, 9.594084644184086, 8.628675143026193, 3.375374771403436, 5.550728329719569, 3.6969899702529068, 9.044565821408213, 4.768291718269253, 4.959443935928806, 2.18487164687376, 6.996568810021718, 7.051580665396808, 2.171949594813306, 6.292094346333709, 4.806480518381338, 5.659382644119237, 2.5496764450570266, 5.7889691512205275, 3.1380286491265568, 2.5470262468096694, 2.372859413494039, 4.231541510471827, 7.999961570513447, 9.432035207279897, 8.280033683080656, 1.010001171327032, 5.938848869894208, 2.514143880766919, 4.527534122871792, 5.790769163566419, 7.243461613521523, 4.910369192202851, 1.3788598341731098, 5.08047217293921, 9.19306972924195, 5.460441982434551, 8.567972131084394, 1.3596790808382135, 4.85673053906484, 5.700585456433414, 9.653533583690477, 6.504693014389554, 8.87117508026694, 3.5671802348746153, 4.616341182550321, 2.2211642155909477, 2.0026264167632473, 6.876871119323232, 5.25559133428939, 9.653424477222297, 5.761175858184941, 1.2912141053395847, 9.786928247232504, 3.521145739469658, 4.716027125558734, 3.057973056972541, 7.611309700108411, 4.320984808114248, 8.907713654459553, 1.195761436491835, 5.981356422710521, 3.900444503618301, 3.6790638406753446, 7.860990692677564, 5.399754599087321, 8.64185633086677, 8.204339107063369, 2.8001847148342067, 7.26753599305138, 6.656979340475203, 4.947911859383337, 6.50855879440713, 2.325183794134949, 7.18787078438455, 8.097916043014, 5.5156225883221115, 2.5183931186161446, 1.5522109221216716, 1.7290363671780198, 4.105066679758549, 1.1990130369571506, 1.721734365455567, 2.8566824698228763, 1.4632288903113762, 6.534070929521084, 4.229487333168938, 8.784494954866155, 9.306319921756842, 1.931163293102339, 4.303726003178453, 4.14267533070238, 6.398335580578855, 9.477257981764348, 1.8979009008393362, 3.8050467348478128, 4.673778851070901, 7.101287486734387, 1.4763758227682249, 8.694158554177648, 1.3163654547466854, 7.596634712991049, 6.210294431567701, 7.4539610936174645, 2.190510576691428, 6.11040002954271, 3.9132732452937953, 1.9121329949669488, 3.860784082321069, 6.888450033965992, 2.770196447299629, 2.2412439233216066, 4.495202359338469, 5.787319560314739, 9.502573174185885, 1.3982198060601827, 2.1847219000292575, 4.0228407253395915, 4.655484184468431, 8.170574541346385, 4.555269441327189, 2.9339129495120044, 5.136438322665492, 8.164990032142322, 1.0511254058227637, 5.845913660152064, 4.701443420948379, 8.999508398705728, 1.3273312432972593, 6.123397184360965, 7.503952303302851, 5.133570602592855, 6.4124148429354655, 3.5416568862557987, 8.77418548414562, 2.4407792590145463, 1.6947945867504428, 3.181121836324218, 5.559282706224457, 8.517976609136891, 2.100941389805768, 7.059958875942, 7.235198743600901, 1.9283786239178007, 6.132244089452554, 9.601422113877948, 1.5767980081706763, 5.3445074045077865, 8.669542819662936, 3.6002199619828317, 7.291214514427308, 4.705830763045604, 4.539914537881251, 5.345618257849709, 2.399840871939055, 2.5912415976787653, 2.3809587386004676, 7.473350716300497, 9.153578348345285, 3.436746494742289, 2.5587947178083583, 5.51244665246702, 7.403224797285749, 5.226094779787628, 1.2393484502701817, 1.6987659293059416, 1.464018958721453, 7.139346036295069, 6.745192643451037, 5.260046591588058, 7.113831972709566, 1.9205795472218128, 2.7527864738207395, 9.718363765953383, 1.6474168004128624, 5.581143156841959, 3.9516958931187416, 9.942920626857555, 1.292958793772259, 5.888636889106974, 1.216472849512812, 9.606566908769228, 3.84721216351457, 5.655390691516903, 5.151856617745188, 6.119178528186397, 3.694119385628807, 8.999232221943668, 7.225503217569478, 8.932293591152348, 5.891757243730633, 5.105770790703005, 5.540836080057998, 4.9597320074056706, 8.569652024246842, 8.458509244248322, 6.09556387097579, 9.420010315677105, 3.785140307678641, 4.471783326698559, 3.2216471504498903, 3.1798812621178416, 4.917405428137128, 2.6042078744991617, 2.103169594620646, 8.765029577079357, 7.789160196734843, 9.541132547549044, 6.317713540053207, 1.0786812008427282, 1.7483065585333897, 5.408951654165381, 6.654268959904409, 5.345928618926037, 5.000261895892443, 5.880827721890511, 7.24793108985076, 8.16649024831171, 6.7107723395771375, 4.683782430851524, 2.837352115633405, 5.412124638579581, 8.355554504499764, 8.90893262443879, 6.154277707130782, 1.557374904314425, 2.2776870736611947, 4.069146771687086, 8.867048705668864, 8.30590377896658, 6.237271555878102, 9.225132300496826, 8.870390982279336, 6.48155362989269, 3.6048954269695126, 1.4669262016593345, 8.825649085629319, 9.641531041786408, 7.355258252540915, 6.502959155757564, 1.5732526753331184, 8.747341469905125, 6.38957304623956, 7.001412690292357, 4.39435964673894, 4.590920461251445, 2.36762462382718, 7.879566202531418, 5.065895480897169, 8.463699201995375, 8.026623076597506, 4.260028963633886, 3.0278578530069153, 9.77149380125834, 8.458244296334026, 3.533773733994953, 9.6581591092709, 6.69165732516654, 9.69333016866568, 4.686116237728703, 4.3466277403818045, 6.977078291950329, 6.7625397743816595, 6.229559362592406, 5.790463871467594, 6.917789337164865, 1.0986502988654412, 4.7390032742695265, 4.501134346187977, 1.8526106479890245, 7.73700633147353, 8.670154185701575, 7.97080557895628, 1.5573059709264543, 2.112633192152836, 8.900084360857107, 8.544362348364512, 5.450616265418107, 2.1408778744191, 4.249822345480317, 7.081858503556818, 1.4900084008052032, 3.2575255882496514, 5.298335568850327, 7.026479274404242, 5.123846612194707, 7.970027817164664, 4.938697342635605, 5.997264732797927, 6.711403873350364, 2.3148705808654944, 4.348075148818628, 8.13701493409211, 9.758546056615337, 2.1082316743372798, 9.956356947505645, 1.3700869849790944, 5.453933638284494, 4.458404048827581, 6.069426725387646, 6.5611769783501845, 6.668635532771923, 3.5051202868676823, 1.5099584034480729, 8.352684498441436, 7.960607601558938, 6.552136907803936, 3.4887860260289694, 2.18968238440204, 8.824836033892435, 1.582800130454774, 1.6681058895559293, 7.163279783570894, 6.296772394524895, 4.640036953700579, 4.416490320921827, 7.691662770397278, 3.085340223977582, 8.192215637862134, 5.518908575576506, 3.638444679677485, 9.410015351253538, 8.64330516699441, 7.763279916798762, 7.912862631642354, 8.557702894152804, 7.343074969739744, 2.845266057735258, 8.044713007852623, 7.321108524587233, 2.952807963589417, 9.987629630573359, 4.019443758513433, 8.679107898167722, 2.304343012115648, 5.185422994039372, 9.846979120873467, 4.1417860918254785, 4.593265947674921, 6.07227998876716, 3.0756115451351844, 6.917749532580873, 6.114722000118023, 9.292180928647367, 1.5450565034807915, 9.907422084838275, 4.710488403344192, 6.865353629996044, 1.848463715166823, 4.089986677780836, 3.293587478846315, 2.2797675093096057, 3.0909908274746503, 2.547341344565217, 6.363895327224248, 6.656552779596629, 8.558788899573107, 7.904714564060619, 3.178364547644724, 3.696106797064396, 2.2749560698026157, 3.3287028749389758, 2.1495124575097493, 4.793580449432424, 3.7512674480847883, 4.156178428072067, 7.832725662972089, 4.534819888852716, 2.8906365007406936, 5.403766261638151, 6.02917902951041, 7.436059286274984, 5.541196245038887, 3.1522929224636616, 4.359643070170151, 8.094249754216879, 8.678002150803055, 1.5203091337979233, 2.306449729143848, 5.7545953155086575, 6.812957871008694, 6.540908809439198, 1.6199384178738225, 2.2909076467042984, 8.958209272274402, 5.948005181904916, 5.515373856236963, 8.845311551618643, 4.392335261913989, 8.88472760667401816, 5.0172233179423795, 9.531788810588033, 2.2222892990766465, 9.382369751452238, 3.899092422800559, 1.5583302362388471, 3.2228797012748527, 5.208558929435964, 2.2216514463922765, 6.7287900768197275, 
3.4163095338661758, 4.5473093565490315, 1.75966869253552, 9.15924116251126, 3.010829207182708, 7.235784270835617, 4.614398407218248, 5.788454687311337, 7.986953943130999, 8.390567962791469, 7.334128360885848, 5.904264179296691, 5.871632241927294, 8.90029094199767, 8.098592778087681, 1.435498485112555, 9.496771918721764, 2.7260960345486707, 6.495834314807186, 8.366578473512549, 4.476146859998354, 3.582249265354218
rand_no0 time :  0.06168007850646973
[5.64763795 6.23777332 4.25516604 ... 7.70816897 2.20661711 6.10630868]
rand_no1 time :  0.0023050308227539062

 

위 출력에서 list는 모든 값을 다 출력해서 처리 시간이 약 0.06초 걸렸지만 numpy 값은 형태만을 간단하게 보여주고 처리시간이 0.002초로 줄어 들었습니다.